Ultimate factors driving insect body size are rather well understood, while—apart from a few model species—the underlying physiological and developmental mechanisms received less attention. We investigate the physiological basis of adaptive size variation in the yellow dung fly Scathophaga stercoraria, which shows pronounced male-biased sexual size dimorphism and strong body size plasticity. We estimate variation of a major physiological threshold, the critical weight, which is the mass at which a larva initiates pupariation. Critical weight was associated with sexual size dimorphism and sex-specific plasticity, and is thus a likely target of selection on adult size. Detailed larval growth trajectories derived from individuals raised at two food and temperature treatments further reveal that sex-specific size plasticity is mediated by faster initial growth of males that later becomes reduced by higher male weight loss during the wandering stage. We further demonstrate that integral growth rates, which are typically calculated as simple ratios of egg-to-adult development time and adult weight, do not necessarily well reflect variation in instantaneous growth rates. We illustrate the importance of detailed assessments of ontogenetic growth trajectories for the understanding of adaptive size variation and discuss the mechanistic basis of size determination in shaping sex-specific phenotypic plasticity.