Effects of low serum 25OHD on age-related changes in muscle mass and function remain unclear. Our aims were to explore associations of baseline 25OHD levels with prevalent and incident sarcopenia and changes in muscle parameters, and to examine the role of parathyroid hormone (PTH) therein. Cross-sectional (n = 975) and prospective analyses (n = 702) of older adults aged 65–93 years participating in the KORA-Age study. Sarcopenia was defined using the 2010 European Working Group on Sarcopenia in Older People (EWGSOP) criteria as low muscle mass combined with low grip strength or low physical performance. Associations with baseline 25OHD were examined in multiple regression analyses. Low vitamin D status was linked to increased odds of prevalent sarcopenia. Over three years, low baseline 25OHD < 25 vs. ≥ 50 nmol/L were associated with greater loss of muscle mass and increased time for the Timed Up and Go test. The risk for developing incident sarcopenia was not significantly elevated in individuals with low baseline 25OHD but when including death as combined outcome alongside incident sarcopenia, there was a strong positive association in multivariable analysis [OR (95% CI) 3.19 (1.54–6.57) for 25OHD < 25 vs. ≥ 50 nmol/L]. There was no evidence for a PTH-mediating effect. Low baseline 25OHD levels were associated with unfavorable changes in muscle mass and physical performance, but not with incident sarcopenia. Future randomized trials are needed to assess causality and to address the issue of competing risks such as mortality in older cohorts.