Header

UZH-Logo

Maintenance Infos

Many analysts, one dataset: Making transparent how variations in analytical choices affect results


Silberzahn, R; Uhlmann, E L; Martin, D P; Anselmi, P; Ullrich, Johannes (2017). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. : PsyArXiv.

Abstract

Abstract Twenty-nine teams involving 61 analysts used the same dataset to address the
same research question: whether soccer referees are more likely to give red cards to dark
skin toned players than light skin toned players. Analytic approaches varied widely across
teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median
of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams
(31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique
combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor
their level of expertise, nor peer-reviewed quality of analysis readily explained variation in
analysis outcomes. This suggests that significant variation in analysis of complex data may
be difficult to avoid, even by experts with honest intentions.

Abstract

Abstract Twenty-nine teams involving 61 analysts used the same dataset to address the
same research question: whether soccer referees are more likely to give red cards to dark
skin toned players than light skin toned players. Analytic approaches varied widely across
teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median
of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams
(31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique
combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor
their level of expertise, nor peer-reviewed quality of analysis readily explained variation in
analysis outcomes. This suggests that significant variation in analysis of complex data may
be difficult to avoid, even by experts with honest intentions.

Statistics

Citations

6 citations in Microsoft Academic

Downloads

41 downloads since deposited on 08 Feb 2018
41 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Scientific Publication in Electronic Form
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Language:English
Date:2017
Deposited On:08 Feb 2018 13:18
Last Modified:31 Jul 2018 04:30
Publisher:PsyArXiv
Number of Pages:79
OA Status:Green
Free access at:Official URL. An embargo period may apply.
Official URL:https://psyarxiv.com/qkwst/

Download

Download PDF  'Many analysts, one dataset: Making transparent how variations in analytical choices affect results'.
Preview
Content: Accepted Version
Filetype: PDF (Version 5)
Size: 4MB
Licence: Creative Commons: Public Domain Dedication: CC0 1.0 Universal (CC0 1.0)