Header

UZH-Logo

Maintenance Infos

Assessing replicability with the sceptical p‐value: Type‐I error control and sample size planning


Micheloud, Charlotte; Balabdaoui, Fadoua; Held, Leonhard (2023). Assessing replicability with the sceptical p‐value: Type‐I error control and sample size planning. Statistica Neerlandica, 77(4):573-591.

Abstract

We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical ‐value. A recalibration is proposed to obtain exact overall Type‐I error control if the effect is null in both studies and additional bounds on the partial and conditional Type‐I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two‐trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two‐trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.

Abstract

We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical ‐value. A recalibration is proposed to obtain exact overall Type‐I error control if the effect is null in both studies and additional bounds on the partial and conditional Type‐I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two‐trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two‐trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.

Statistics

Citations

Dimensions.ai Metrics
2 citations in Web of Science®
2 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

4 downloads since deposited on 23 Jan 2024
4 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Epidemiology, Biostatistics and Prevention Institute (EBPI)
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Physical Sciences > Statistics and Probability
Social Sciences & Humanities > Statistics, Probability and Uncertainty
Uncontrolled Keywords:Statistics, Probability and Uncertainty, Statistics and Probability
Language:English
Date:November 2023
Deposited On:23 Jan 2024 12:18
Last Modified:30 Jun 2024 01:37
Publisher:Wiley-Blackwell Publishing, Inc.
ISSN:0039-0402
OA Status:Hybrid
Free access at:Publisher DOI. An embargo period may apply.
Publisher DOI:https://doi.org/10.1111/stan.12312
Project Information:
  • : FunderSchweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
  • : Grant ID
  • : Project Title
  • Content: Published Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)