Header

UZH-Logo

Maintenance Infos

The “Goldilocks Zone”: (Too) many confidence intervals in tests of mediation just exclude zero


Götz, Martin; O'Boyle, Ernest H; Gonzalez-Mulé, Erik; Banks, George C; Bollmann, Stella S (2021). The “Goldilocks Zone”: (Too) many confidence intervals in tests of mediation just exclude zero. Psychological Bulletin, 147(1):95-114.

Abstract

Questionable research practices (QRPs) can occur whenever one result is favored over another, and tests of mediation are no exception. Given mediation's ubiquity and importance to both theory and practice, QRPs in tests of mediation pose a serious threat to the advancement of psychology. We investigate this issue through the introduction of a straightforward means of detecting the presence and magnitude of QRPs in tests of mediation and validate this methodology with a series of sensitivity tests and simulations. We then apply this method to 2,569 tests of mediation published in five leading psychology journals in 2018 and 2019. We find that despite most hypothesized tests of mediation being likely underpowered, most (76%) were nevertheless supported. Furthermore, confidence intervals (CIs) that just barely exclude zero are 3.6 to 4.4 times as prevalent as those CIs that just barely include zero. We also find a number of study- and test-level factors, such as whether the test of mediation was hypothesized, explain both whether the CI excluded zero (odds ratio [OR] = 17.87, p < .001) as well as the CI's proximity to zero (b = .27, p < .001). In addition, other factors, most notably sample size, do predict the CI's proximity to zero (γ = .00, p < .001), but surprisingly do not predict the CI's exclusion of zero (OR = .99, p = .803). We conclude with actionable QRP curtailment strategies so that both, academics and practitioners, can have greater and well-founded confidence in tests of mediation in psychological research.

Abstract

Questionable research practices (QRPs) can occur whenever one result is favored over another, and tests of mediation are no exception. Given mediation's ubiquity and importance to both theory and practice, QRPs in tests of mediation pose a serious threat to the advancement of psychology. We investigate this issue through the introduction of a straightforward means of detecting the presence and magnitude of QRPs in tests of mediation and validate this methodology with a series of sensitivity tests and simulations. We then apply this method to 2,569 tests of mediation published in five leading psychology journals in 2018 and 2019. We find that despite most hypothesized tests of mediation being likely underpowered, most (76%) were nevertheless supported. Furthermore, confidence intervals (CIs) that just barely exclude zero are 3.6 to 4.4 times as prevalent as those CIs that just barely include zero. We also find a number of study- and test-level factors, such as whether the test of mediation was hypothesized, explain both whether the CI excluded zero (odds ratio [OR] = 17.87, p < .001) as well as the CI's proximity to zero (b = .27, p < .001). In addition, other factors, most notably sample size, do predict the CI's proximity to zero (γ = .00, p < .001), but surprisingly do not predict the CI's exclusion of zero (OR = .99, p = .803). We conclude with actionable QRP curtailment strategies so that both, academics and practitioners, can have greater and well-founded confidence in tests of mediation in psychological research.

Statistics

Citations

Dimensions.ai Metrics
29 citations in Web of Science®
30 citations in Scopus®
Google Scholar™

Altmetrics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:06 Faculty of Arts > Institute of Psychology
Dewey Decimal Classification:150 Psychology
Scopus Subject Areas:Social Sciences & Humanities > General Psychology
Uncontrolled Keywords:History and Philosophy of Science, General Psychology
Language:English
Date:31 December 2021
Deposited On:11 Jan 2021 15:42
Last Modified:25 Mar 2024 02:45
Publisher:American Psychological Association
ISSN:0033-2909
OA Status:Closed
Publisher DOI:https://doi.org/10.1037/bul0000315
Full text not available from this repository.