Header

UZH-Logo

Maintenance Infos

Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results


Silberzahn, R; Uhlmann, E L; Martin, D P; et al; Ullrich, Johannes (2017). Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. PsyArXiv qkwst.

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions.

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions.

Statistics

Citations

Dimensions.ai Metrics

6 citations in Microsoft Academic

Altmetrics

Downloads

171 downloads since deposited on 08 Feb 2018
141 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Working Paper
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Language:English
Date:2017
Deposited On:08 Feb 2018 13:18
Last Modified:29 Mar 2019 08:03
Series Name:PsyArXiv
Number of Pages:79
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Publisher DOI:https://doi.org/10.31234/osf.io/qkwst
Official URL:https://psyarxiv.com/qkwst/
Related URLs:https://doi.org/10.1177/2515245917747646 (Publisher)
https://www.zora.uzh.ch/id/eprint/159498/

Download

Download PDF  'Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results'.
Preview
Content: Accepted Version
Filetype: PDF (Version 5)
Size: 4MB
View at publisher
Licence: Creative Commons: Public Domain Dedication: CC0 1.0 Universal (CC0 1.0)