The additive genetic variation (VA) of fitness in a population is of particular importance to quantify its adaptive potential and predict its response to rapid environmental change. Recent statistical advances in quantitative genetics and the use of new molecular tools have fostered great interest in estimating fitness VA in wild populations. However, the value of VA for fitness in predicting evolutionary changes over several generations remains mostly unknown. In our study, we addressed this question by combining classical quantitative genetics with experimental evolution in the model organism Tribolium castaneum (red flour beetle) in three new environmental conditions (Dry, Hot, Hot‐Dry). We tested for potential constraints that might limit adaptation, including environmental and sex genetic antagonisms captured by negative genetic covariance between environments and female and male fitness, respectively. Observed fitness changes after 20 generations mainly matched our predictions. Given that body size is commonly used as a proxy for fitness, we also tested how this trait and its genetic variance (including nonadditive genetic variance) were impacted by environmental stress. In both traits, genetic variances were sex and condition dependent, but they differed in their variance composition, cross‐sex and cross‐environment genetic covariances, as well as in the environmental impact on VA.