Binomial confidence intervals for rare events: importance of defining margin of error relative to magnitude of proportion

Confidence interval performance is typically assessed in terms of two criteria: coverage probability and interval width (or margin of error). In this work we detail the importance of defining the margin of error in relation to the magnitude of the estimated proportion when the success probability is small. We compare the performance of four common proportion interval estimators: the Wald, Clopper-Pearson, Wilson and Agresti-Coull, in the context of rare-event probabilities. We show that incompatibilities between the margin of error and the magnitude of the proportion results in very narrow confidence intervals (requiring extremely large sample sizes), or intervals that are too wide to be practically useful. We propose a relative margin of error scheme that is consistent with the order of magnitude of the proportion. Confidence interval performance is thus assessed in terms of satisfying this relative margin of error, in conjunction with achieving a desired coverage probability. We illustrate that when adherence to this relative margin of error is considered as a requirement for satisfactory interval performance, all four interval estimators perform somewhat similarly for a given sample size and confidence level.