Multiple testing corrections for p-values and confidence intervals from generalised linear mixed models

In this article, we describe and compare methods to derive \textit{p}-values and sets of confidence intervals with strong control of the family-wise error rates and coverage for estimates of treatment effects in cluster randomised trials with multiple outcomes. While the need for corrections for multiple testing is debated, the justification for doing so is that, without correction, the probability of rejecting at least one of a set of null hypotheses is greater than the nominal rate of any single test, and hence the coverage of a confidence set is lower than the nominal rate of any single interval. There are few methods \textit{p}-value corrections in this context and none for deriving confidence intervals, limiting their application in this setting. We discuss the methods of Bonferroni, Holm, and Romano \& Wolf (2005) and their adaptation to cluster randomised trials using permutational inference using efficient quasi-score test statistics and weighted generalised residuals. We develop a search procedure for confidence interval limits using permutation tests to produce a set of confidence intervals under each method of correction. We conduct a simulation-based study to compare family-wise error rates, coverage of confidence sets, and the efficiency of each procedure in comparison to no correction using both model-based standard errors and permutation tests. We show that the Romano-Wolf type procedure has nominal error rates and coverage under non-independent correlation structures and is more efficient than the other methods in a simulation-based study. We also compare results from the analysis of a real-world trial.