Measuring Evidence with a Continuous Test

Testing has developed into the fundamental statistical framework for falsifying hypotheses. Unfortunately, tests are binary in nature: a test either rejects a hypothesis or not. Such binary decisions do not reflect the reality of many scientific studies, which often aim to present the evidence against a hypothesis and do not necessarily intend to establish a definitive conclusion. To solve this, we propose the continuous generalization of a test, which we use to measure the evidence against a hypothesis. Such a continuous test can be viewed as a continuous non-randomized interpretation of the classical 'randomized test'. This offers the benefits of a randomized test, without the downsides of external randomization. Another interpretation is as a literal measure, which measures the amount of binary tests that reject the hypothesis. Our work completes the bridge between classical tests and the recently proposed $e$-values: $e$-values bounded to $[0, 1/\alpha]$ are continuously interpreted size $\alpha$ randomized tests. Taking $\alpha$ to 0 yields the regular $e$-value: a 'level 0' continuous test. Moreover, we generalize the traditional notion of power by using generalized means. This produces a unified framework that contains both classical Neyman-Pearson optimal testing and log-optimal $e$-values, as well as a continuum of other options. The traditional $p$-value appears as the reciprocal of an $e$-value, that satisfies a weaker error bound. In an illustration in a Gaussian location model, we find that optimal continuous tests are of a beautifully simple form.