Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Schweinsberg, Martin; Feldman, Michael; Staub, Nicola; van den Akker, Olmo R; van Aert, Robbie C M; van Assen, Marcel A L M; Liu, Yang; Althoff, Tim; Heer, Jeffrey; Kale, Alex; Mohamed, Zainab; Amireh, Hashem; Venkatesh Prasad, Vaishali; Bernstein, Abraham; Robinson, Emily; Snellman, Kaisa; Amy Sommer, S; Otner, Sarah M G; Robinson, David; Madan, Nikhil

doi:10.5167/uzh-205202

Citations

Citation copied

Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A., Mohamed, Z., Amireh, H., Venkatesh Prasad, V., Bernstein, A., Robinson, E., Snellman, K., Amy Sommer, S., Otner, S. M. G., Robinson, D., … Luis Uhlmann, E. (2021). Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes, 165, 228–249. https://doi.org/10.1016/j.obhdp.2021.02.003

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic