Post-hoc Valid Inference

A pervasive methodological error is the post-hoc interpretation of $p$-values. A $p$-value $p$ is not the level at which we reject the null, it is the level at which we would have rejected the null had we chosen level $p$. We introduce the notion of a post-hoc $p$-value, that does admit this interpretation. We show that $p$ is a post-hoc $p$-value if and only if $1/p$ is an $e$-value. Among other things, this implies that the product of independent post-hoc $p$-values is a post-hoc $p$-value. Moreover, we generalize post-hoc validity to a sequential setting and find that $(p_t)_{t \geq 1}$ is a post-hoc anytime valid $p$-process if and only if $(1/p_t)_{t \geq 1}$ is an $e$-process. In addition, we show that if we admit randomized procedures, any non-randomized post-hoc $p$-value can be trivially improved. In fact, we find that this in some sense characterizes non-randomized post-hoc $p$-values. Finally, we argue that we need to go beyond $e$-values if we want to consider randomized post-hoc inference in its full generality.