Binomial confidence intervals for rare events: importance of defining margin of error relative to magnitude of proportion

Confidence interval performance is typically assessed in terms of two criteria: coverage probability and interval width (or margin of error). In this work we assess the performance of four common proportion interval estimators: the Wald, Clopper- Pearson, Wilson and Agresti-Coull, in the context of rare-event probabilities. We define the precision of the interval estimate in terms of a relative margin of error which ensures consistency with the magnitude of the proportion. Thus, confidence interval performance is assessed in terms of achieving a desired coverage probability whilst satisfying the specified relative margin of error. We show that when interval performance is considered using both coverage probability and relative margin of error, all four interval estimators perform somewhat similarly for a given sample size and confidence level. We identify relative margin of error values that result in satisfactory coverage whilst being conservative in terms of sample size requirements, and hence suggest a range of values that can be adopted in practice. The proposed relative margin of error scheme is evaluated analytically, by simulation, and by application to a number of recent studies from the literature.