Remote imaging spectroscopy in the 0.4–2.5-μm visible and shortwave infrared (VSWIR) range captures the majority of solar-reflected energy and enables a wide range of earth surface studies. This spectral range is also influenced by atmospheric effects including absorption from atmospheric gases and aerosols, Rayleigh scattering, and particle scattering. Globally consistent surface measurements must compensate for these atmospheric effects. This article reviews the physical and mathematical foundations of modern VSWIR atmospheric retrieval, focusing on imaging spectrometers. We assess sensitivity of the retrieval to errors in atmospheric state estimation. Finally, we describe some promising avenues of future research to support the next generation of orbital imaging spectrometers.