This study was conducted to examine the statistical influence of brain size on cortical, subcortical, and cerebellar compartmental volumes. This brain size influence was especially studied to delineate interactions with Sex and Age. Here, we studied 856 healthy subjects of which 533 are classified as young and 323 as old. Using an automated segmentation procedure cortical (gray and white matter [GM and WM] including the corpus callosum), cerebellar (GM and WM), and subcortical (thalamus, putamen, pallidum, caudatus, hippocampus, amygdala, and accumbens) volumes were measured and subjected to statistical analyses. These analyses revealed that brain size and age exert substantial statistical influences on nearly all compartmental volumes. Analyzing the raw compartmental volumes replicated the frequently reported Sex differences in compartmental volumes with men showing larger volumes. However, when statistically controlling for brain size Sex differences and Sex × Age interactions practically disappear. Thus, brain size is more important than Sex in explaining interindividual differences in compartmental volumes. The influence of brain size is discussed in the context of an allometric scaling of the compartmental volumes.