Header

UZH-Logo

Maintenance Infos

Reproducibility in the absence of selective reporting: An illustration from large‐scale brain asymmetry research


Kong, Xiang‐Zhen; Francks, Clyde; Mathias, Samuel R; Guadalupe, Tulio; et al; Walitza, Susanne (2020). Reproducibility in the absence of selective reporting: An illustration from large‐scale brain asymmetry research. Human Brain Mapping:Epub ahead of print.

Abstract

The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left-right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an "ideal publishing environment," that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.

Abstract

The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p-hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left-right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta-analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an "ideal publishing environment," that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically-used sample sizes.

Statistics

Citations

Altmetrics

Downloads

7 downloads since deposited on 24 Sep 2020
7 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Psychiatric University Hospital Zurich > Department of Child and Adolescent Psychiatry
04 Faculty of Medicine > Neuroscience Center Zurich
04 Faculty of Medicine > Center for Integrative Human Physiology
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Health Sciences > Anatomy
Health Sciences > Radiological and Ultrasound Technology
Health Sciences > Radiology, Nuclear Medicine and Imaging
Life Sciences > Neurology
Health Sciences > Neurology (clinical)
Uncontrolled Keywords:Anatomy, Radiological and Ultrasound Technology, Radiology Nuclear Medicine and imaging, Neurology, Clinical Neurology
Language:English
Date:25 August 2020
Deposited On:24 Sep 2020 12:28
Last Modified:02 Oct 2020 12:30
Publisher:Wiley-Blackwell Publishing, Inc.
ISSN:1065-9471
OA Status:Hybrid
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.1002/hbm.25154
PubMed ID:32841457

Download

Hybrid Open Access

Download PDF  'Reproducibility in the absence of selective reporting: An illustration from large‐scale brain asymmetry research'.
Preview
Content: Published Version
Language: English
Filetype: PDF
Size: 1MB
View at publisher
Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)