Healthy aging is associated with changes in cognitive performance and functional brain organization. In fact, cross-sectional studies imply lower modularity and significant heterogeneity in modular architecture across older subjects. Here, we used a longitudinal dataset consisting of four occasions of resting-state-fMRI and cognitive testing (spanning 4 years) in 150 healthy older adults. We applied a graph-theoretic analysis to investigate the time-evolving modular structure of the whole-brain network, by maximizing the multilayer modularity across four time points. Global flexibility, which reflects the tendency of brain nodes to switch between modules across time, was significantly higher in healthy elderly than in a temporal null model. Further, global flexibility, as well as network-specific flexibility of the default mode, frontoparietal control, and somatomotor networks, were significantly associated with age at baseline. These results indicate that older age is related to higher variability in modular organization. The temporal metrics were not associated with simultaneous changes in processing speed or learning performance in the context of memory encoding. Finally, this approach provides global indices for longitudinal change across a given time span and it may contribute to uncovering patterns of modular variability in healthy and clinical aging populations.