On the Second-Order Asymptotics of the Hoeffding Test and Other Divergence Tests

TL;DR

This work analyzes binary hypothesis testing with known null $P$ and unknown alternative $Q$, focusing on the Hoeffding test and a broader class of divergence tests ${ ext T}_n^D(r)$. It derives exact first- and second-order asymptotics for the type-II error under a fixed type-I constraint, showing that all divergence tests share the same first-order term $D_{ ext{KL}}(P\, Vert Q)$ as the Neyman-Pearson test, but their second-order terms depend on the chosen divergence. Invariant divergences yield the same second-order term as the Hoeffding test, while non-invariant divergences can outperform Hoeffding for some $Q$, indicating potential benefits for composite tests with partial knowledge of the alternative. The results are supported by a generalized chi-square tail approximation and extensive numerical experiments illustrating when different tests have the best finite-sample performance and how the cardinality of the alphabet impacts scalability.

Abstract

Consider a binary statistical hypothesis testing problem, where $n$ independent and identically distributed random variables $Z^n$ are either distributed according to the null hypothesis $P$ or the alternative hypothesis $Q$, and only $P$ is known. A well-known test that is suitable for this case is the so-called Hoeffding test, which accepts $P$ if the Kullback-Leibler (KL) divergence between the empirical distribution of $Z^n$ and $P$ is below some threshold. This work characterizes the first and second-order terms of the type-II error probability for a fixed type-I error probability for the Hoeffding test as well as for divergence tests, where the KL divergence is replaced by a general divergence. It is demonstrated that, irrespective of the divergence, divergence tests achieve the first-order term of the Neyman-Pearson test, which is the optimal test when both $P$ and $Q$ are known. In contrast, the second-order term of divergence tests is strictly worse than that of the Neyman-Pearson test. It is further demonstrated that divergence tests with an invariant divergence achieve the same second-order term as the Hoeffding test, but divergence tests with a non-invariant divergence may outperform the Hoeffding test for some alternative hypotheses $Q$. Potentially, this behavior could be exploited by a composite hypothesis test with partial knowledge of the alternative hypothesis $Q$ by tailoring the divergence of the divergence test to the set of possible alternative hypotheses.

Figures (8)

• Figure 1: Inverse tail probability functions of the Normal and chi-square distributions as a function of $\epsilon$.
• Figure 2: Ratio $\rho(P,Q,\epsilon)$ of the second-order terms of the Hoeffding test ${\mathsf{T}}^{D_{\textnormal{KL}}}_n(r)$ and the divergence test ${\mathsf{T}}^{D_{\textnormal{SM}}}_n(r)$ as a function of $\mathbf{Q}$ for $\epsilon=0.02$.
• Figure 3: Ratio $\rho(P,Q,\epsilon)$ of the second-order terms of the Hoeffding test ${\mathsf{T}}^{D_{\textnormal{KL}}}_n(r)$ and the divergence test ${\mathsf{T}}^{D_{\textnormal{SM}}}_n(r)$ as a function of $Q_1$ for $Q_2=0.5$ and $\epsilon=0.02$.
• Figure 4: Ratio $\rho(P,Q,\epsilon)$ of the second-order terms of the Hoeffding test ${\mathsf{T}}^{D_{\textnormal{KL}}}_n(r)$ and the divergence test ${\mathsf{T}}^{D_{\textnormal{SM}}}_n(r)$ as a function of $\mathbf{Q}$ for $P=(0.1,0.3,0.6)$.
• Figure 5: Type-I error vs. type-II error for $P=(0.15, 0.6, 0.25)$ and $n=500$.

Theorems & Definitions (25)

• Definition 1
• Remark 2
• Definition 3
• Remark 4
• Definition 5
• Lemma 6
• proof
• Theorem 7
• proof
• Remark 8