Header

UZH-Logo

Maintenance Infos

The statistical properties of RCTs and a proposal for shrinkage


van Zwet, Erik; Schwab, Simon; Senn, Stephen (2021). The statistical properties of RCTs and a proposal for shrinkage. Statistics in Medicine, 40(27):6107-6117.

Abstract

We abstract the concept of a randomized controlled trial as a triple (β,b,s) , where β is the primary efficacy parameter, b the estimate, and s the standard error ( s>0 ). If the parameter β is either a difference of means, a log odds ratio or a log hazard ratio, then it is reasonable to assume that b is unbiased and normally distributed. This then allows us to estimate the joint distribution of the z-value z=b/s and the signal-to-noise ratio SNR=β/s from a sample of pairs (bi,si) . We have collected 23 551 such pairs from the Cochrane database. We note that there are many statistical quantities that depend on (β,b,s) only through the pair (z,SNR) . We start by determining the estimated distribution of the achieved power. In particular, we estimate the median achieved power to be only 13%. We also consider the exaggeration ratio which is the factor by which the magnitude of β

is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so. We show that our shrinkage estimator successfully addresses the exaggeration. As an example, we re-analyze the ANDROMEDA-SHOCK trial.

Keywords: Cochrane review; achieved power; exaggeration; randomized controlled trial; type M error.

Abstract

We abstract the concept of a randomized controlled trial as a triple (β,b,s) , where β is the primary efficacy parameter, b the estimate, and s the standard error ( s>0 ). If the parameter β is either a difference of means, a log odds ratio or a log hazard ratio, then it is reasonable to assume that b is unbiased and normally distributed. This then allows us to estimate the joint distribution of the z-value z=b/s and the signal-to-noise ratio SNR=β/s from a sample of pairs (bi,si) . We have collected 23 551 such pairs from the Cochrane database. We note that there are many statistical quantities that depend on (β,b,s) only through the pair (z,SNR) . We start by determining the estimated distribution of the achieved power. In particular, we estimate the median achieved power to be only 13%. We also consider the exaggeration ratio which is the factor by which the magnitude of β

is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so. We show that our shrinkage estimator successfully addresses the exaggeration. As an example, we re-analyze the ANDROMEDA-SHOCK trial.

Keywords: Cochrane review; achieved power; exaggeration; randomized controlled trial; type M error.

Statistics

Citations

Dimensions.ai Metrics
13 citations in Web of Science®
14 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

170 downloads since deposited on 21 Sep 2021
49 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:04 Faculty of Medicine > Epidemiology, Biostatistics and Prevention Institute (EBPI)
04 Faculty of Medicine > Center for Reproducible Science
Dewey Decimal Classification:610 Medicine & health
Scopus Subject Areas:Health Sciences > Epidemiology
Physical Sciences > Statistics and Probability
Uncontrolled Keywords:Statistics and Probability, Epidemiology
Language:English
Date:30 November 2021
Deposited On:21 Sep 2021 12:54
Last Modified:25 Jun 2024 01:44
Publisher:Wiley-Blackwell Publishing, Inc.
ISSN:0277-6715
OA Status:Hybrid
Free access at:PubMed ID. An embargo period may apply.
Publisher DOI:https://doi.org/10.1002/sim.9173
PubMed ID:34425632
  • Content: Published Version
  • Licence: Creative Commons: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)