Header

UZH-Logo

Maintenance Infos

Responsible Research Assessment Should Prioritize Theory Development and Testing Over Ticking Open Science Boxes


Dames, Hannah; Musfeld, Philipp; Popov, Vencislav; Oberauer, Klaus; Frischkorn, Gidon T (2023). Responsible Research Assessment Should Prioritize Theory Development and Testing Over Ticking Open Science Boxes. PsyArXiv Preprints ad74m, Cornell University.

Abstract

We appreciate the initiative to seek for ways to improve academic assessment by broadening the range of relevant research contributions and by considering a candidate’s scientific rigor. Evaluating a candidate's ability to contribute to science is a complex process that cannot be captured through one metric alone. While the proposed changes have some advantages, such as an increased focus on quality over quantity, the proposal's focus on adherence to open science practices is not sufficient, as it undervalues theory building and formal modelling: A narrow focus on open science conventions is neither a sufficient nor valid indicator for a “good scientist” and may even encourage researchers to choose easy, pre-registerable studies rather than engage in time-intensive theory building. Further, when in a first step only a minimum standard for following easily achievable open science goals is set, most applicants will soon pass this threshold. At this point, one may ask if the additional benefit of such a low bar outweighs the potential costs of such an endeavour. We conclude that a reformed assessment system should put at least equal emphasis on theory building and adherence to open science principles and should not completely disregard traditional performance metrices.

Abstract

We appreciate the initiative to seek for ways to improve academic assessment by broadening the range of relevant research contributions and by considering a candidate’s scientific rigor. Evaluating a candidate's ability to contribute to science is a complex process that cannot be captured through one metric alone. While the proposed changes have some advantages, such as an increased focus on quality over quantity, the proposal's focus on adherence to open science practices is not sufficient, as it undervalues theory building and formal modelling: A narrow focus on open science conventions is neither a sufficient nor valid indicator for a “good scientist” and may even encourage researchers to choose easy, pre-registerable studies rather than engage in time-intensive theory building. Further, when in a first step only a minimum standard for following easily achievable open science goals is set, most applicants will soon pass this threshold. At this point, one may ask if the additional benefit of such a low bar outweighs the potential costs of such an endeavour. We conclude that a reformed assessment system should put at least equal emphasis on theory building and adherence to open science principles and should not completely disregard traditional performance metrices.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

53 downloads since deposited on 09 May 2023
54 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Working Paper
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Language:English
Date:14 March 2023
Deposited On:09 May 2023 10:40
Last Modified:17 May 2023 09:48
Series Name:PsyArXiv Preprints
ISSN:0010-9452
OA Status:Green
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.31234/osf.io/ad74m
  • Content: Accepted Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)