Header

UZH-Logo

Maintenance Infos

A new standard for the analysis and design of replication studies


Held, Leonhard (2019). A new standard for the analysis and design of replication studies. arXiv.org 1811.10287, Cornell University.

Abstract

A new standard is proposed for the evidential assessment of replication studies. The approach combines a specific reverse-Bayes technique with prior-predictive tail probabilities to define replication success. The method gives rise to a quantitative measure for rep- lication success, called the skeptical p-value. The skeptical p -value integrates traditional significance of both the original and replication study with a comparison of the respective effect sizes. It incorporates the uncertainty of both the original and replication effect estimates and reduces to the ordinary p -value of the replication study if the uncertainty of the original effect estimate is ignored. The proposed framework can also be used to determine the power or the required replication sample size to achieve replication success. Numerical cal- culations highlight the difficulty to achieve replication success if the evidence from the original study is only suggestive. An application to data from the Open Science Collaboration project on the replicability of psychological science illustrates the proposed methodology.

Abstract

A new standard is proposed for the evidential assessment of replication studies. The approach combines a specific reverse-Bayes technique with prior-predictive tail probabilities to define replication success. The method gives rise to a quantitative measure for rep- lication success, called the skeptical p-value. The skeptical p -value integrates traditional significance of both the original and replication study with a comparison of the respective effect sizes. It incorporates the uncertainty of both the original and replication effect estimates and reduces to the ordinary p -value of the replication study if the uncertainty of the original effect estimate is ignored. The proposed framework can also be used to determine the power or the required replication sample size to achieve replication success. Numerical cal- culations highlight the difficulty to achieve replication success if the evidence from the original study is only suggestive. An application to data from the Open Science Collaboration project on the replicability of psychological science illustrates the proposed methodology.

Statistics

Downloads

31 downloads since deposited on 04 Oct 2019
31 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Working Paper
Communities & Collections:04 Faculty of Medicine > Epidemiology, Biostatistics and Prevention Institute (EBPI)
Dewey Decimal Classification:610 Medicine & health
Language:English
Date:14 May 2019
Deposited On:04 Oct 2019 15:04
Last Modified:07 Apr 2020 07:22
Series Name:arXiv.org
Number of Pages:32
ISSN:2331-8422
OA Status:Green
Official URL:https://arxiv.org/pdf/1811.10287.pdf

Download

Green Open Access

Download PDF  'A new standard for the analysis and design of replication studies'.
Preview
Content: Published Version
Filetype: PDF
Size: 487kB