Header

UZH-Logo

Maintenance Infos

Does evaluation quality enhance evaluation use?


Bundi, Pirmin; Frey, Kathrin; Widmer, Thomas (2021). Does evaluation quality enhance evaluation use? Evidence & Policy, 17(4):661-687.

Abstract

Background: Evaluations are a useful tool to learn more about the effectiveness of public measures. In the era of evidence-based policymaking, recent research suggests that quality is an important determinant of the utilisation of evaluations. Despite this claim, hardly any empirical study has investigated whether the quality of an evaluation – measured by a meta-evaluation – influences its perceived utilisation by decision makers.Aims and objectives: This article asks how the quality of an evaluation study is related to its perceived use, and investigates the relationship between the quality of an evaluation, assessed through a meta-evaluation, and how the evaluation is perceived and accepted by the parties concerned.Methods: The basis for the empirical analyses were 34 external evaluations, conducted from 2006 to 2014, of upper secondary schools in the canton of Zurich, as well as a standardised survey conducted among 307 representatives of these schools (teachers, administrators, members of quality development teams, and the heads of school oversight commissions).Findings: We conclude that the quality of the evaluation, as assessed in a meta-evaluation, is not particularly associated with the perception of evaluation quality and the perceived use of the evaluation. The perceived quality, however, is related to the perceived impact of an evaluation.Discussion and conclusion: These findings are relevant for evaluation research and practice, since they show that the quality of an evaluation and evaluation use do not necessarily go hand in hand.<br />Key messages<br /><ul><li>Evaluators have to be aware that a systematically assessed quality of an evaluation does not go hand in hand with the perceived quality of that evaluation;</li><br /><li>Evaluators often focus on the instrumental form of evaluation use, but they should not ignore other forms of use and maybe try to maximise these utilisation forms in the design of their evaluation;</li><br /><li>Evaluators should be more active in advising stakeholders when it comes to evaluation use, for example, through policy narratives;</li><br /><li>Evaluators should carefully think about the measurement of evaluation quality and evaluation effects in research on evaluation.</li></ul>

Abstract

Background: Evaluations are a useful tool to learn more about the effectiveness of public measures. In the era of evidence-based policymaking, recent research suggests that quality is an important determinant of the utilisation of evaluations. Despite this claim, hardly any empirical study has investigated whether the quality of an evaluation – measured by a meta-evaluation – influences its perceived utilisation by decision makers.Aims and objectives: This article asks how the quality of an evaluation study is related to its perceived use, and investigates the relationship between the quality of an evaluation, assessed through a meta-evaluation, and how the evaluation is perceived and accepted by the parties concerned.Methods: The basis for the empirical analyses were 34 external evaluations, conducted from 2006 to 2014, of upper secondary schools in the canton of Zurich, as well as a standardised survey conducted among 307 representatives of these schools (teachers, administrators, members of quality development teams, and the heads of school oversight commissions).Findings: We conclude that the quality of the evaluation, as assessed in a meta-evaluation, is not particularly associated with the perception of evaluation quality and the perceived use of the evaluation. The perceived quality, however, is related to the perceived impact of an evaluation.Discussion and conclusion: These findings are relevant for evaluation research and practice, since they show that the quality of an evaluation and evaluation use do not necessarily go hand in hand.<br />Key messages<br /><ul><li>Evaluators have to be aware that a systematically assessed quality of an evaluation does not go hand in hand with the perceived quality of that evaluation;</li><br /><li>Evaluators often focus on the instrumental form of evaluation use, but they should not ignore other forms of use and maybe try to maximise these utilisation forms in the design of their evaluation;</li><br /><li>Evaluators should be more active in advising stakeholders when it comes to evaluation use, for example, through policy narratives;</li><br /><li>Evaluators should carefully think about the measurement of evaluation quality and evaluation effects in research on evaluation.</li></ul>

Statistics

Citations

Dimensions.ai Metrics
2 citations in Web of Science®
5 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

81 downloads since deposited on 01 Apr 2021
34 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Political Science
Dewey Decimal Classification:320 Political science
Uncontrolled Keywords:social sciences (miscellaneous), physiology, orthopedics and sports medicine, cardiology and cardiovascular medicine evaluation quality, evaluation use, meta-evaluation, survey
Language:English
Date:2021
Deposited On:01 Apr 2021 08:51
Last Modified:26 Mar 2024 02:35
Publisher:Policy Press
ISSN:1744-2648
OA Status:Green
Publisher DOI:https://doi.org/10.1332/174426421x16141794148067
  • Content: Accepted Version
  • Language: English