On admissibility in post-hoc hypothesis testing
Ben Chugg, Tyron Lardy, Aaditya Ramdas, Peter Grünwald
TL;DR
The paper develops a principled framework for post-hoc hypothesis testing, where the significance level $\alpha$ can be data-dependent and is governed by risk-based losses rather than fixed error probabilities. Central to the approach is $\Gamma$-admissibility, which compares test families against adversaries mapping data to losses; this yields a rich theory connecting test design to e-values. It extends the Neyman–Pearson paradigm by showing how likelihood-ratio-type procedures generalize to post-hoc settings and by providing sharp, canonical representations of admissible tests via e-values. The work further characterizes admissible rules for two adversary classes, $\mathbf{U}$ and $\mathbf{C}$, recovering the Neyman–Pearson lemma in the constant-maps limit and highlighting Rao–Blackwellization as a general tool for improving tests. Overall, the results bridge traditional hypothesis testing with modern decision-theoretic and e-value methods, offering data-dependent guarantees and guiding practical post-hoc inference.
Abstract
The validity of classical hypothesis testing requires the significance level $α$ be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating $α$ during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value $p\llα$ vs $p= α- ε$ for tiny $ε> 0$ has no (statistical) relevance for any downstream decision-making. Following recent work of Grünwald (2024), we develop a theory of post-hoc hypothesis testing, enabling $α$ to be chosen after seeing and analyzing the data. To study "good" post-hoc tests we introduce $Γ$-admissibility, where $Γ$ is a set of adversaries which map the data to a significance level. We classify the set of $Γ$-admissible rules for various sets $Γ$, showing they must be based on e-values, and recover the Neyman-Pearson lemma when $Γ$ is the constant map.