The statistical properties of RCTs and a proposal for shrinkage

We abstract the concept of a randomized controlled trial (RCT) as a triple (beta,b,s), where beta is the primary efficacy parameter, b the estimate and s the standard error (s>0). The parameter beta is either a difference of means, a log odds ratio or a log hazard ratio. If we assume that b is unbiased and normally distributed, then we can estimate the full joint distribution of (beta,b,s) from a sample of pairs (b_i,s_i). We have collected 23,747 such pairs from the Cochrane database to do so. Here, we report the estimated distribution of the signal-to-noise ratio beta/s and the achieved power. We estimate the median achieved power to be 0.13. We also consider the exaggeration ratio which is the factor by which the magnitude of beta is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so.