Safe Testing

We develop the theory of hypothesis testing based on the E-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on E-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO E-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO E-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test (in which the right Haar prior turns out to be GRO) and the 2x2 contingency table (in which the GRO prior is different from standard priors). Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, E-values and the corresponding tests may provide a methodology acceptable to adherents of all three schools.