The generation of well-calibrated radiometric measurements from imaging spectrometer data requires careful consideration of all influencing factors, as well as an instrument calibration based on a detailed sensor model. Deviations of ambient parameters (i.e., pressure, humidity, temperature) from standard laboratory conditions during airborne operations can lead to biases that should be accounted for and properly compensated by using dedicated instrument models. This study introduces a model for the airborne imaging spectrometer airborne prism experiment (APEX), describing the impact of spectral shifts as well as polarization effects on the radiometric system response due to changing ambient parameters. Key issues are related to changing properties of the dichroic coating applied to the dispersing elements within the optical path. We present a model based on discrete numerical simulations. With the improved modeling approach, we predict radiometric biases with an root mean square error (RMSE) below 1%, leading to a substantial improvement of radiometric stability and predictability of system behavior.