Header

UZH-Logo

Maintenance Infos

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis


Schweinsberg, Martin; Feldman, Michael; Staub, Nicola; van den Akker, Olmo R; van Aert, Robbie C M; van Assen, Marcel A L M; Liu, Yang; Althoff, Tim; Heer, Jeffrey; Kale, Alex; Mohamed, Zainab; Amireh, Hashem; Venkatesh Prasad, Vaishali; Bernstein, Abraham; Robinson, Emily; Snellman, Kaisa; Amy Sommer, S; Otner, Sarah M G; Robinson, David; Madan, Nikhil; Silberzahn, Raphael; Goldstein, Pavel; Tierney, Warren; Murase, Toshio; Mandl, Benjamin; Viganola, Domenico; Strobl, Carolin; Schaumans, Catherine B C; Kelchtermans, Stijn; Naseeb, Chan; Mason Garrison, S; Yarkoni, Tal; Richard Chan, C S; Adie, Prestone; Alaburda, Paulius; Albers, Casper; Alspaugh, Sara; Alstott, Jeff; Nelson, Andrew A; Ariño de la Rubia, Eduardo; Arzi, Adbi; Bahník, Štěpán; Baik, Jason; Winther Balling, Laura; Banker, Sachin; AA Baranger, David; Barr, Dale J; Barros-Rivera, Brenda; Bauer, Matt; Blaise, Enuh; Boelen, Lisa; Bohle Carbonell, Katerina; Briers, Robert A; Burkhard, Oliver; Canela, Miguel-Angel; Castrillo, Laura; Catlett, Timothy; Chen, Olivia; Clark, Michael; Cohn, Brent; Coppock, Alex; Cugueró-Escofet, Natàlia; Curran, Paul G; Cyrus-Lai, Wilson; Dai, David; Valentino Dalla Riva, Giulio; Danielsson, Henrik; Russo, Rosaria de F S M; de Silva, Niko; Derungs, Curdin; Dondelinger, Frank; Duarte de Souza, Carolina; Tyson Dube, B; Dubova, Marina; Mark Dunn, Ben; Adriaan Edelsbrunner, Peter; Finley, Sara; Fox, Nick; Gnambs, Timo; Gong, Yuanyuan; Grand, Erin; Greenawalt, Brandon; Han, Dan; Hanel, Paul H P; Hong, Antony B; Hood, David; Hsueh, Justin; Huang, Lilian; Hui, Kent N; Hultman, Keith A; Javaid, Azka; Ji Jiang, Lily; Jong, Jonathan; Kamdar, Jash; Kane, David; Kappler, Gregor; Kaszubowski, Erikson; Kavanagh, Christopher M; Khabsa, Madian; Kleinberg, Bennett; Kouros, Jens; Krause, Heather; Krypotos, Angelos-Miltiadis; Lavbič, Dejan; Ling Lee, Rui; Leffel, Timothy; Yang Lim, Wei; Liverani, Silvia; Loh, Bianca; Lønsmann, Dorte; Wei Low, Jia; Lu, Alton; MacDonald, Kyle; Madan, Christopher R; Hjorth Madsen, Lasse; Maimone, Christina; Mangold, Alexandra; Marshall, Adrienne; Ester Matskewich, Helena; Mavon, Kimia; McLain, Katherine L; McNamara, Amelia A; McNeill, Mhairi; Mertens, Ulf; Miller, David; Moore, Ben; Moore, Andrew; Nantz, Eric; Nasrullah, Ziauddin; Nejkovic, Valentina; Nell, Colleen S; Arthur Nelson, Andrew; Nilsonne, Gustav; Nolan, Rory; O'Brien, Christopher E; O'Neill, Patrick; O'Shea, Kieran; Olita, Toto; Otterbacher, Jahna; Palsetia, Diana; Pereira, Bianca; Pozdniakov, Ivan; Protzko, John; Reyt, Jean-Nicolas; Riddle, Travis; (Akmal) Ridhwan Omar Ali, Amal; Ropovik, Ivan; Rosenberg, Joshua M; Rothen, Stephane; Schulte-Mecklenbeck, Michael; Sharma, Nirek; Shotwell, Gordon; Skarzynski, Martin; Stedden, William; Stodden, Victoria; Stoffel, Martin A; Stoltzman, Scott; Subbaiah, Subashini; Tatman, Rachael; Thibodeau, Paul H; Tomkins, Sabina; Valdivia, Ana; Druijff-van de Woestijne, Gerrieke B; Viana, Laura; Villesèche, Florence; Duncan Wadsworth, W; Wanders, Florian; Watts, Krista; Wells, Jason D; Whelpley, Christopher E; Won, Andy; Wu, Lawrence; Yip, Arthur; Youngflesh, Casey; Yu, Ju-Chi; Zandian, Arash; Zhang, Leilei; Zibman, Chava; Luis Uhlmann, Eric (2021). Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes, 165:228-249.

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

Statistics

Citations

Dimensions.ai Metrics
46 citations in Web of Science®
48 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

90 downloads since deposited on 27 Jul 2021
11 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:03 Faculty of Economics > Department of Informatics
08 Research Priority Programs > Digital Society Initiative
Dewey Decimal Classification:000 Computer science, knowledge & systems
Scopus Subject Areas:Social Sciences & Humanities > Applied Psychology
Social Sciences & Humanities > Organizational Behavior and Human Resource Management
Language:English
Date:2021
Deposited On:27 Jul 2021 08:40
Last Modified:25 Feb 2024 02:41
Publisher:Elsevier
ISSN:0749-5978
OA Status:Hybrid
Publisher DOI:https://doi.org/10.1016/j.obhdp.2021.02.003
Official URL:https://www.sciencedirect.com/science/article/pii/S0749597821000200
Other Identification Number:merlin-id:21275
  • Content: Published Version
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)