Associations between leisure time physical activity (LTPA) and all-cause mortality seem quite strong, however, less is known about the association of LTPA and cause-specific mortality. To examine this association data from the Third National Health and Nutrition Examination Survey (NHANES III), including 15,307 individuals of the non-institutionalized civilian United States population, were used. Data were collected from 1988 to 1994 with a mortality follow-up until 2006. LTPA was assessed during home interviews in which participants specified their LTPA and the performed frequency during the past month. Cox proportional hazards regression models were applied to analyze the risk of cause-specific mortality regarding LTPA. Hazard ratios (HRs) and 95% confidence intervals (CIs) were computed. An inverse association of LTPA with CVD mortality was observed for men and women combined for irregular (HR 0.66; 95% CI 0.51–0.85), and for regular activity (HR 0.58; 95% CI 0.47–0.72). An inverse association of LTPA with CVD mortality was observed only in women for irregular (HR 0.64; 95% CI 0.49–0.84) and for regular activity (HR 0.55; 95% CI 0.43–0.72). In men, no significant associations were seen. For mortality caused by respiratory diseases, a decreased mortality was also observed in the combined group (men and women) but after separating according to sex a decreased mortality was only observed in women. No statistically significant association of LTPA with cancer mortality was observed. Our data support an inverse association between LTPA and CVD and respiratory disease mortality in women, but not in men, and no associations with cancer.