Doublethink: simultaneous Bayesian-frequentist model-averaged hypothesis testing

Establishing the frequentist properties of Bayesian approaches widens their appeal and offers new understanding. In hypothesis testing, Bayesian model averaging addresses the problem that conclusions are sensitive to variable selection. But Bayesian false discovery rate (FDR) guarantees are contingent on prior assumptions that may be disputed. Here we show that Bayesian model-averaged hypothesis testing is a closed testing procedure that controls the frequentist familywise error rate (FWER) in the strong sense. The rate converges pointwise as the sample size grows and, under some conditions, uniformly. The `Doublethink' method computes simultaneous posterior odds and asymptotic p-values for model-averaged hypothesis testing. We explore its benefits, including post-hoc variable selection, and limitations, including finite-sample inflation, through a Mendelian randomization study and simulations comparing approaches like LASSO, stepwise regression, the Benjamini-Hochberg procedure and e-values.