Gravitational lensing time delays are well known to depend on cosmological parameters, but they also depend on the details of the mass distribution of the lens. It is usual to model the mass distribution and use time-delay observations to infer cosmological parameters, but it is naturally also possible to take the cosmological parameters as given and use time delays as constraints on the mass distribution. In this paper we develop a method of isolating what exactly those constraints are, using a principal-components analysis of ensembles of free-form mass models. We find that time delays provide tighter constraints on the distribution of matter in the very highly dense regions of the lensing clusters. We apply it to the cluster lens SDSS J1004+4112, whose rich lensing data include two time delays. We find, assuming a concordance cosmology, that the time delays constrain the central region of the cluster to be rounder and less lopsided than would be allowed by lensed images alone. This detailed information about the distribution of the matter is very useful for studying the dense regions of the galaxy clusters which are very difficult to study by direct measurements. A further time-delay measurement, which is expected, will make this system even more interesting.