With the help of a Markov State Model (MSM), two-state behaviour is resolved for two computer models of water in a temperature range from 255 K to room temperature (295 K). The method is first validated for ST2 water, for which the so far strongest evidence for a liquid-liquid phase transition exists. In that case, the results from the MSM can be cross-checked against the radial distribution function g(5)(r) of the 5th-closest water molecule around a given reference water molecule. The latter is a commonly used local order parameter, which exhibits a bimodal distribution just above the liquid-liquid critical point that represents the low-density form of the liquid (LDL) and the high density liquid. The correlation times and correlation lengths of the corresponding spatial domains are calculated and it is shown that they are connected via a simple diffusion model. Once the approach is established, TIP4P/2005 will be considered, which is the much more realistic representation of real water. The MSM can resolve two-state behavior also in that case, albeit with significantly smaller correlation times and lengths. The population of LDL-like water increases with decreasing temperature, thereby explaining the density maximum at 4 degrees C along the lines of the two-state model of water. Published by AIP Publishing.