It has long been proposed that cognitive aging in fluid abilities is driven by age-related declines of processing speed. Although study of between-person associations generally supports this view, accumulating longitudinal between-person and within-person evidence indicates less strong associations between speed and fluid cognitive performance. Initial evidence also suggests that cognitive flexibility may explain within-person variability in cognitive performance. In the present study, we used up to nine waves of data over 56 years from a subsample of 582 participants of the Seattle Longitudinal Study to examine (a) within-person associations of psychomotor speed and cognitive flexibility with cognitive aging in primary mental abilities (including inductive reasoning, number ability, verbal meaning, spatial orientation, and word fluency); and (b) how these within-person associations change with age. In line with the processing speed theory, results revealed that within persons, primary mental abilities (including fluid, crystallized, and visualization measures) were indeed associated with psychomotor speed. We also observed age-related increases in within-person couplings between primary mental abilities and psychomotor speed. While the processing speed theory focuses primarily on associations with fluid abilities, age-related increases in coupling were found for a variety of ability domains. Within-person associations between primary mental abilities and cognitive flexibility were weaker and relatively stable with age. We discuss the role of speed and flexibility for cognitive aging.