Bayesian Nonparametric Sensitivity Analysis of Multiple Test Procedures Under Dependence

This article introduces a sensitivity analysis method for Multiple Testing Procedures (MTPs) using marginal $p$-values. The method is based on the Dirichlet process (DP) prior distribution, specified to support the entire space of MTPs, where each MTP controls either the family-wise error rate (FWER) or the false discovery rate (FDR) under arbitrary dependence between $p$-values. The DP MTP sensitivity analysis method accounts for uncertainty in the selection of such MTPs and their respective cut-off points and decisions regarding which subset of $p$-values are significant from a given set of hypothesis tested, while measuring each $p$-value's probability of significance over the DP prior predictive distribution of this space of all MTPs, and reducing the possible conservativeness of using one such MTP for multiple testing. The DP MTP sensitivity analysis method is illustrated through the analysis of twenty-eight thousand $p$-values arising from hypothesis tests performed on a 2022 dataset of a representative sample of three million U.S. high school students observed on 239 variables. They include tests that relate variables about the disruption caused by school closures during the COVID-19 pandemic, with variables on mathematical cognition and academic achievement, and with student background variables. R software code for the DP MTP sensitivity analysis method is provided in the Appendix and in Supplementary Information.