Coherent phonons are an excellent tool to investigate the interplay between electronic and structural dynamics. The displacive excitation of coherent phonons in elemental bismuth is one of the most widely studied processes for this purpose. We employ time-resolved photoelectron diffraction to access the structural dynamics by recording the photoemission intensity from one initial state as a function of emission angle. In comparison with tight-binding and single-scattering cluster calculations, this allows electronic and structural effects to be disentangled. Hence, the full dynamics of the hot electron gas and of coherently excited phonons can be accessed in a single experiment. As a major result the phase lag between the coherent phonons and the modulation of the electronic structure can be determined with high precision. The phonon phase lag with respect to the modulation of the electronic structure is about 2.85±0.21 rad, thus significantly smaller than π. The difference is not due to phonon decay by energy dissipation into low-energy modes, but rather caused by the very early evolution of the highly excited electron distribution.