We investigate the correlation of gravitational lensing of the cosmic microwave background (CMB) with several tracers of large-scale structure, including luminous red galaxies (LRGs), quasars, and radio sources. The lensing field is reconstructed based on the CMB maps from the Wilkinson Microwave Anisotropy Probe (WMAP) satellite; the LRGs and quasars are observed by the Sloan Digital Sky Survey (SDSS); and the radio sources are observed in the NRAO VLA Sky Survey (NVSS). Combining all three large-scale structure samples, we find evidence for a positive cross-correlation at the $2.5\sigma$ level ($1.8\sigma$ for the SDSS samples and $2.1\sigma$ for NVSS); the cross-correlation amplitude is $1.06\pm 0.42$ times that expected for the WMAP cosmological parameters. Our analysis extends other recent analyses in that we carefully determine bias weighted redshift distribution of the sources, which is needed for a meaningful cosmological interpretation of the detected signal. We investigate contamination of the signal by Galactic emission, extragalactic radio and infrared sources, thermal and kinetic Sunyaev-Zel'dovich effects, and the Rees-Sciama effect, and find all of them to be negligible.