Crowd‐based hydrological observations can supplement existing monitoring networks and allow data collection in regions where otherwise no data would be available. In the citizen science project CrowdWater, repeated water level observations using a virtual staff gauge approach result in time series of water level classes (WL‐classes). To investigate the quality of these observations, we compared the WL‐class data with “real” (i.e., measured) water levels from the same stream at a nearby gauging station. We did this for nine locations where citizen scientists reported multiple observations using a smartphone app and at 12 locations where signposts were set up to ask citizens to record observations on a paper form that could be left in a letterbox. The results indicate that the quality of the data collected with the app was better than for the forms. A possible explanation is that for each app location, a single person submitted the vast majority of the observations, whereas at the locations of the forms almost every observation was made by a different person. On average, there were more contributions between May and September than during the other months. Observations were submitted for a range of flow conditions, with a higher fraction of high flow observations for the locations were data were collected with the app. Overall, the results are encouraging for citizen science approaches in hydrology and demonstrate that the smartphone application and the virtual staff gauge are a promising approach for crowd‐based water level class observations.