Research is increasingly focusing on promoting healthy aging and the related extension of the health span by targeting crucial biological processes responsible for age-related conditions. While age-related gradual changes in steroid hormones such as testosterone, estradiol, or cortisol are well described in men, their interactions among each other or with genetic markers have not been sufficiently investigated with regard to physical health or psychological well-being. More specifically, the examination of age-related alterations in hormone interactions and the androgen receptor polymorphism, which modulates androgen action on target cells, in relation to physical health and psychological well-being represents a promising avenue for research on healthy aging in men. A total of 97 healthy aging men provided complete data on psychometric health measures as well as hormonal and genetic parameters at baseline and a 4-year follow-up assessment. Fasting saliva samples were taken at 8:00 am under standardized laboratory conditions, while the androgen receptor gene polymorphism was analyzed from dried blood spots. Longitudinal analyses revealed that psychological well-being and physical health remained stable over time. Analyses indicated that E2 moderated the course of psychological well-being, while the androgen receptor gene polymorphism moderated the course of physical health. Further, T was a strong predictor of physical health. These results suggest that the hypothalamic-pituitary-gonadal (HPG) axis might be important for the maintenance of psychological well-being in men, while physical health depends more on interindividual differences in the androgen receptor gene and T.