Header

UZH-Logo

Maintenance Infos

Many analysts, one dataset: Making transparent how variations in analytical choices affect results


Silberzahn, R; Uhlmann, E L; Martin, D P; et al; Ullrich, Johannes (2017). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. PsyArXiv qkwst.

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions.

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions.

Statistics

Citations

Dimensions.ai Metrics

6 citations in Microsoft Academic

Altmetrics

Downloads

156 downloads since deposited on 08 Feb 2018
155 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Working Paper
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Language:English
Date:2017
Deposited On:08 Feb 2018 13:18
Last Modified:11 Dec 2018 15:28
Series Name:PsyArXiv
Number of Pages:79
Additional Information:Peer-reviewed Publication DOI: 10.1177/2515245917747646
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Publisher DOI:https://doi.org/10.31234/osf.io/qkwst
Official URL:https://psyarxiv.com/qkwst/
Related URLs:https://doi.org/10.1177/2515245917747646 (Publisher)

Download

Download PDF  'Many analysts, one dataset: Making transparent how variations in analytical choices affect results'.
Preview
Content: Accepted Version
Filetype: PDF (Version 5)
Size: 4MB
View at publisher
Licence: Creative Commons: Public Domain Dedication: CC0 1.0 Universal (CC0 1.0)