The assessment of replication success based on relative effect size

Abstract

Replication studies are increasingly conducted to confirm original findings. However, there is no established standard how to assess replication success and in practice many different approaches are used. The purpose of this paper is to refine and extend a recently proposed reverse-Bayes approach for the analysis of replication studies. We show how this method is directly related to the relative effect size, the ratio of the replication to the original effect estimate. This perspective leads to two important contributions: the golden level to recalibrate the assessment of replication success and a novel approach to calculate the replication sample size based on the specification of the minimum relative effect size. Compared to the standard approach to require statistical significance of both the original and replication study, replication success at the golden level offers uniform gains in project power and controls the Type-I error rate even if the replication sample size is slightly smaller than the original one. Sample size calculation based on replication success at the golden level tends to require smaller samples than the standard approach, if the original study is reasonably powered. An application to data from four large replication projects shows that the replication success approach leads to more appropriate inferences, as it penalizes shrinkage of the replication estimate compared to the original one, while ensuring that both effect estimates are sufficiently convincing on their own.