Seeds are multi-generational structures containing a small embryonic plant enclosed in layers of diverse parental origins. The evolution of seeds was a pinnacle in an evolutionary trend towards a progressive retention of embryos and gametes within parental tissue. This strategy, which dates back to the first land plants, allowed an increased protection and nourishing of the developing embryo. Flowering plants took parental control one step further with the evolution of a biparental endosperm that derives from a second parallel fertilization event. The endosperm directly nourishes the developing embryo and allows not only the maternal genes, but also paternal genes, to play an active role during seed development. The appearance of an endosperm set the conditions for the manifestation of conflicts of interest between maternal and paternal genomes over the allocation of resources to the developing embryos. As a consequence, a dynamic balance was established between maternal and paternal gene dosage in the endosperm, and maintaining a correct balance became essential to ensure a correct seed development. This balance was achieved in part by changes in the genetic constitution of the endosperm and through epigenetic mechanisms that allow a differential expression of alleles depending on their parental origin. This review discusses the evolutionary steps that resulted in the appearance of seeds and endosperm, and the epigenetic and genetic mechanisms that allow a harmonious coinhabitance of multiple generations within a single seed.