Header

UZH-Logo

Maintenance Infos

Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results


Silberzahn, R; Uhlmann, E L; Martin, D P; et al; Ullrich, Johannes (2017). Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. PsyArXiv Preprints qkwst, PsyArXiv.

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions.

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions.

Statistics

Citations

Dimensions.ai Metrics

Altmetrics

Downloads

212 downloads since deposited on 08 Feb 2018
7 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Working Paper
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Language:English
Date:2017
Deposited On:08 Feb 2018 13:18
Last Modified:24 Jan 2024 15:26
Series Name:PsyArXiv Preprints
Number of Pages:79
ISSN:0010-9452
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Publisher DOI:https://doi.org/10.31234/osf.io/qkwst
Official URL:https://psyarxiv.com/qkwst/
Related URLs:https://doi.org/10.1177/2515245917747646 (Publisher)
https://www.zora.uzh.ch/id/eprint/159498/
  • Content: Accepted Version
  • Description: Version 5
  • Licence: Creative Commons: Public Domain Dedication: CC0 1.0 Universal (CC0 1.0)