Header

UZH-Logo

Maintenance Infos

Special on Survey Methodology: What is the price of going fully online? A mixed-mode sampling experiment on the occasion of the Swiss general elections 2019


Milic, Thomas; Serdült, Uwe; Brüggemann, Salim (2020). Special on Survey Methodology: What is the price of going fully online? A mixed-mode sampling experiment on the occasion of the Swiss general elections 2019. Aarau, Switzerland: Zentrum für Demokratie Aarau (ZDA).

Abstract

The FOKUS Aargau Mixed Mode Survey Experiment

Survey-based research is currently in the process of moving partly or completely online. The scalability of online surveys not only helps to cut costs but also allows for a streamlined production of reporting which draws directly on the data. Data pre-processing can be cut to a minimum or be largely eliminated. Whereas current experience with online surveys in Switzerland shows good response rates, less is known about the quality of the sample in comparison to telephone or postal surveys. On the occasion of the October 2019 general elections in Switzerland, the FOKUS Aargau team at the ZDA therefore conducted an experiment evaluating how the response rate as well as the sample quality might differ in an online versus an offline survey. Based on the official address register, 10'000 randomly drawn residents of the Canton of Aargau who are also Swiss citizens received an invitation letter asking them to take part in the survey. While half of the sample received a paper version as well as a pre-paid envelope, giving them the option to submit their questionnaires by mail, the other half was directed to the address of the online survey with a provided link (or QR-code) only.

Results

The initial as well as total response rate among survey respondents who had the choice of either using the paper version and sending it in via pre-paid mail or using the online version was somewhat higher than for the sample in which the only available channel with which to respond was the internet. However, the difference in response rates between the two modes is not large and one might conclude that the paper mode is more or less obsolete. However, response rates are not the only criterion when evaluating the quality of a survey sample. With regard to representativeness the two sub-samples tend to complement each other. The sample with the mail-in option is more representative of gender, older age groups and people with a lower formal level of education. For the online sample the pattern is the reverse. In total, the mixed-mode samples, in combination, clearly yielded the most representative sample. Regarding over-reporting, there is no difference between the two survey modes. The most striking differences between respondents' answering mode are revealed when looking at the political preferences, which are crucial for a post-election survey. Mail-in survey respondents clearly tend to favor identifying and voting for the SVP (Swiss People's Party), online mode respondents more often vote for left-wing political parties. However, when controlling for other variables in a multi-variate model, party-based differences almost disappear and what emerges in its place is a pattern of digital divide, e.g. with respondents favoring the nationwide introduction of internet voting tending to opt for the online survey mode. In sum, consistent with other studies, we find that the exclusion of offline households produces significant coverage biases, while the inclusion of these households in the sample improves the representativeness of the survey significantly. Non-response as the last of the three quality criteria compared among the two survey modes showed stark differences. With a meagre completion rate of 52 percent, the paper version fares much worse than the online mode for this survey (84% completion).

Abstract

The FOKUS Aargau Mixed Mode Survey Experiment

Survey-based research is currently in the process of moving partly or completely online. The scalability of online surveys not only helps to cut costs but also allows for a streamlined production of reporting which draws directly on the data. Data pre-processing can be cut to a minimum or be largely eliminated. Whereas current experience with online surveys in Switzerland shows good response rates, less is known about the quality of the sample in comparison to telephone or postal surveys. On the occasion of the October 2019 general elections in Switzerland, the FOKUS Aargau team at the ZDA therefore conducted an experiment evaluating how the response rate as well as the sample quality might differ in an online versus an offline survey. Based on the official address register, 10'000 randomly drawn residents of the Canton of Aargau who are also Swiss citizens received an invitation letter asking them to take part in the survey. While half of the sample received a paper version as well as a pre-paid envelope, giving them the option to submit their questionnaires by mail, the other half was directed to the address of the online survey with a provided link (or QR-code) only.

Results

The initial as well as total response rate among survey respondents who had the choice of either using the paper version and sending it in via pre-paid mail or using the online version was somewhat higher than for the sample in which the only available channel with which to respond was the internet. However, the difference in response rates between the two modes is not large and one might conclude that the paper mode is more or less obsolete. However, response rates are not the only criterion when evaluating the quality of a survey sample. With regard to representativeness the two sub-samples tend to complement each other. The sample with the mail-in option is more representative of gender, older age groups and people with a lower formal level of education. For the online sample the pattern is the reverse. In total, the mixed-mode samples, in combination, clearly yielded the most representative sample. Regarding over-reporting, there is no difference between the two survey modes. The most striking differences between respondents' answering mode are revealed when looking at the political preferences, which are crucial for a post-election survey. Mail-in survey respondents clearly tend to favor identifying and voting for the SVP (Swiss People's Party), online mode respondents more often vote for left-wing political parties. However, when controlling for other variables in a multi-variate model, party-based differences almost disappear and what emerges in its place is a pattern of digital divide, e.g. with respondents favoring the nationwide introduction of internet voting tending to opt for the online survey mode. In sum, consistent with other studies, we find that the exclusion of offline households produces significant coverage biases, while the inclusion of these households in the sample improves the representativeness of the survey significantly. Non-response as the last of the three quality criteria compared among the two survey modes showed stark differences. With a meagre completion rate of 52 percent, the paper version fares much worse than the online mode for this survey (84% completion).

Statistics

Altmetrics

Additional indexing

Item Type:Published Research Report
Communities & Collections:06 Faculty of Arts > Institute of Political Science
02 Faculty of Law > Centre for Democracy Studies Aarau (C2D)
08 Research Priority Programs > Digital Society Initiative
Working Paper Series > FOKUS Aargau
Dewey Decimal Classification:320 Political science
Uncontrolled Keywords:mixed-mode sampling experiment, Wahlen Oktober 2019
Language:English
Date:February 2020
Deposited On:23 Mar 2020 11:32
Last Modified:21 Sep 2020 15:24
Publisher:Zentrum für Demokratie Aarau (ZDA)
Series Name:FOKUS Aargau
Number of Pages:30
ISSN:2624-7399
ISBN:978-3-906918-09-9
OA Status:Closed
Free access at:Official URL. An embargo period may apply.
Official URL:https://fokus.ag/FOKUS_Aargau_Special_on_Survey_Methodology_February_2020.pdf
Related URLs:https://fokus.ag/special_2/

Download

Full text not available from this repository.
Get full-text in a library