Sexual selection can displace traits acting as ornaments or armaments from their viability optimum in one sex, ultimately giving rise to sexual dimorphism. The degree of dimorphism should not only mirror the strength of sexual selection but also the net viability costs of trait maintenance at equilibrium. As the ability of organisms to bear exaggerated traits will depend on their condition, more sexually dimorphic traits should also exhibit greater sex differences in condition dependence. While this has been demonstrated among traits within species, similar patterns are expected across the phylogeny. We investigated this prediction within and across 11 (sub)species of sepsid flies with varying mating systems. When estimating condition dependence for seven sexual and nonsexual traits that vary in their sexual dimorphism, we not only found a positive relationship between the sex difference in allometric slopes (our measure of condition dependence) and relative trait exaggeration within species but also across species for those traits expected to be under sexual selection. Species with more pronounced male aggression further had relatively larger and more condition-dependent male fore- and midlegs. Our comparative study suggests a common genetic/developmental basis of sexual dimorphism and sex-specific plasticity that evolves across the phylogeny—and that the evolution of size consistently alters scaling relationships and thus contributes to the allometric variation of sexual armaments or ornaments in animals.