The effect of estimating prevalences on the population-wise error rate

The population-wise error rate (PWER) is a type I error rate for clinical trials with multiple target populations. In such trials, one treatment is tested for its efficacy in each population. The PWER is defined as the probability that a randomly selected, future patient will be exposed to an inefficient treatment based on the study results. The PWER can be understood and computed as an average of strata-specific family-wise error rates and involves the prevalences of these strata. A major issue of this concept is that the prevalences are usually not known in practice, so that the PWER cannot be directly controlled. Instead, one could use an estimator based on the given sample, like their maximum-likelihood estimator under a multinomial distribution. In this paper we show in simulations that this does not substantially inflate the true PWER. We differentiate between the expected PWER, which is almost perfectly controlled, and study-specific values of the PWER which are conditioned on all subgroup sample sizes and vary within a narrow range. Thereby, we consider up to eight different overlapping patient populations and moderate to large sample sizes. In these settings, we also consider the maximum strata-wise family-wise error rate, which is found to be, on average, at least bounded by twice the significance level used for PWER control. Finally, we introduce an adjustment of the PWER that could be made when, by chance, no patients are recruited from a stratum, so that this stratum is not counted in PWER control. We would then reduce the PWER in order to control for multiplicity in this stratum as well.