Header

UZH-Logo

Maintenance Infos

The assessment of intrinsic credibility and a new argument for p < 0.005


Held, Leonhard (2019). The assessment of intrinsic credibility and a new argument for p < 0.005. Royal Society Open Science, 6(3):181534.

Abstract

The concept of intrinsic credibility has been recently introduced to check the credibility of ‘out of the blue’ findings without any prior support. A significant result is deemed intrinsically credible if it is in conflict with a sceptical prior derived from the very same data that would make the effect just non-significant. In this paper, I propose to use Bayesian prior-predictive tail probabilities to assess intrinsic credibility. For the standard 5% significance level, this leads to a new p-value threshold that is remarkably close to the recently proposed p < 0.005 standard. I also introduce the credibility ratio, the ratio of the upper to the lower limit (or vice versa) of a confidence interval for a significant effect size. I show that the credibility ratio has to be smaller than 5.8 such that a significant finding is also intrinsically credible. Finally, a p-value for intrinsic credibility is proposed that is a simple function of the ordinary p-value and has a direct frequentist interpretation in terms of the probability of replicating an effect. An application to data from the Open Science Collaboration study on the reproducibility of psychological science suggests that intrinsic credibility of the original experiment is better suited to predict the success of a replication experiment than standard significance.

Abstract

The concept of intrinsic credibility has been recently introduced to check the credibility of ‘out of the blue’ findings without any prior support. A significant result is deemed intrinsically credible if it is in conflict with a sceptical prior derived from the very same data that would make the effect just non-significant. In this paper, I propose to use Bayesian prior-predictive tail probabilities to assess intrinsic credibility. For the standard 5% significance level, this leads to a new p-value threshold that is remarkably close to the recently proposed p < 0.005 standard. I also introduce the credibility ratio, the ratio of the upper to the lower limit (or vice versa) of a confidence interval for a significant effect size. I show that the credibility ratio has to be smaller than 5.8 such that a significant finding is also intrinsically credible. Finally, a p-value for intrinsic credibility is proposed that is a simple function of the ordinary p-value and has a direct frequentist interpretation in terms of the probability of replicating an effect. An application to data from the Open Science Collaboration study on the reproducibility of psychological science suggests that intrinsic credibility of the original experiment is better suited to predict the success of a replication experiment than standard significance.

Statistics

Citations

Dimensions.ai Metrics
1 citation in Web of Science®
3 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

19 downloads since deposited on 04 Oct 2019
19 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Epidemiology, Biostatistics and Prevention Institute (EBPI)
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Health Sciences > Multidisciplinary
Language:English
Date:13 March 2019
Deposited On:04 Oct 2019 14:15
Last Modified:22 Apr 2020 21:13
Publisher:Royal Society Publishing
ISSN:2054-5703
OA Status:Gold
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.1098/rsos.181534

Download

Gold Open Access

Download PDF  'The assessment of intrinsic credibility and a new argument for p < 0.005'.
Preview
Content: Published Version
Filetype: PDF
Size: 635kB
View at publisher
Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)