Header

UZH-Logo

Maintenance Infos

The sceptical Bayes factor for the assessment of replication success


Pawel, Samuel; Held, Leonhard (2022). The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society: Series B, 84(3):879-911.

Abstract

Replication studies are increasingly conducted but there is no established statistical criterion for replication success. We propose a novel approach combining reverse-Bayes analysis with Bayesian hypothesis testing: a sceptical prior is determined for the effect size such that the original finding is no longer convincing in terms of a Bayes factor. This prior is then contrasted to an advocacy prior (the reference posterior of the effect size based on the original study), and replication success is declared if the replication data favour the advocacy over the sceptical prior at a higher level than the original data favoured the sceptical prior over the null hypothesis. The sceptical Bayes factor is the highest level where replication success can be declared. A comparison to existing methods reveals that the sceptical Bayes factor combines several notions of replicability: it ensures that both studies show sufficient evidence against the null and penalises incompatibility of their effect estimates. Analysis of asymptotic properties and error rates, as well as case studies from the Social Sciences Replication Project show the advantages of the method for the assessment of replicability.

Abstract

Replication studies are increasingly conducted but there is no established statistical criterion for replication success. We propose a novel approach combining reverse-Bayes analysis with Bayesian hypothesis testing: a sceptical prior is determined for the effect size such that the original finding is no longer convincing in terms of a Bayes factor. This prior is then contrasted to an advocacy prior (the reference posterior of the effect size based on the original study), and replication success is declared if the replication data favour the advocacy over the sceptical prior at a higher level than the original data favoured the sceptical prior over the null hypothesis. The sceptical Bayes factor is the highest level where replication success can be declared. A comparison to existing methods reveals that the sceptical Bayes factor combines several notions of replicability: it ensures that both studies show sufficient evidence against the null and penalises incompatibility of their effect estimates. Analysis of asymptotic properties and error rates, as well as case studies from the Social Sciences Replication Project show the advantages of the method for the assessment of replicability.

Statistics

Citations

Dimensions.ai Metrics
8 citations in Web of Science®
9 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

45 downloads since deposited on 12 Jan 2023
14 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Epidemiology, Biostatistics and Prevention Institute (EBPI)
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Physical Sciences > Statistics and Probability
Social Sciences & Humanities > Statistics, Probability and Uncertainty
Uncontrolled Keywords:Statistics, Probability and Uncertainty, Statistics and Probability
Language:English
Date:1 July 2022
Deposited On:12 Jan 2023 11:44
Last Modified:28 Jun 2024 01:38
Publisher:Wiley-Blackwell Publishing, Inc.
ISSN:1369-7412
OA Status:Green
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.1111/rssb.12491
Related URLs:https://www.zora.uzh.ch/id/eprint/196009/
  • Content: Published Version
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)
  • Content: Accepted Version