Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotically Optimal Tests for One- and Two-Sample Problems

Arick Grootveld, Biao Chen, Venkata Gandikota

TL;DR

The paper analyzes asymptotically optimal tests for binary hypothesis testing when distributions are unknown, focusing on finite-alphabet problems. It presents a streamlined proof that Hoeffding's likelihood ratio test is asymptotically optimal for the one-sample (goodness-of-fit) problem, casting the test as a threshold on the relative entropy between the empirical and nominal distributions $D\left(\hat Q_{X^n}\|P\right)$. This approach extends naturally to the two-sample problem, showing that a threshold on the relative entropy between the two empirical distributions $D\left(\hat P_{X^n}\|\hat Q_{Y^n}\right)$ is also asymptotically optimal, with the optimal error exponent given by $D\left(F^*\|P\right)+D\left(F^*\|Q\right)$, where $F^* = \arg\min_F D(F\|P)+D(F\|Q)$, and equivalently by $2D_{1/2}(P,Q)$ for equally sized samples. The authors also prove a strong converse: any test achieving a faster decay of the type II error must incur a type I error that tends to one. The results hinge on Sanov's theorem, KL-continuity bounds for finite alphabets, and Renyi-divergence connections, yielding a clear, hands-on framework for universal hypothesis testing with finite support.

Abstract

In this work, we revisit the one- and two-sample testing problems: binary hypothesis testing in which one or both distributions are unknown. For the one-sample test, we provide a more streamlined proof of the asymptotic optimality of Hoeffding's likelihood ratio test, which is equivalent to the threshold test of the relative entropy between the empirical distribution and the nominal distribution. The new proof offers an intuitive interpretation and naturally extends to the two-sample test where we show that a similar form of Hoeffding's test, namely a threshold test of the relative entropy between the two empirical distributions is also asymptotically optimal. A strong converse for the two-sample test is also obtained.

Asymptotically Optimal Tests for One- and Two-Sample Problems

TL;DR

The paper analyzes asymptotically optimal tests for binary hypothesis testing when distributions are unknown, focusing on finite-alphabet problems. It presents a streamlined proof that Hoeffding's likelihood ratio test is asymptotically optimal for the one-sample (goodness-of-fit) problem, casting the test as a threshold on the relative entropy between the empirical and nominal distributions . This approach extends naturally to the two-sample problem, showing that a threshold on the relative entropy between the two empirical distributions is also asymptotically optimal, with the optimal error exponent given by , where , and equivalently by for equally sized samples. The authors also prove a strong converse: any test achieving a faster decay of the type II error must incur a type I error that tends to one. The results hinge on Sanov's theorem, KL-continuity bounds for finite alphabets, and Renyi-divergence connections, yielding a clear, hands-on framework for universal hypothesis testing with finite support.

Abstract

In this work, we revisit the one- and two-sample testing problems: binary hypothesis testing in which one or both distributions are unknown. For the one-sample test, we provide a more streamlined proof of the asymptotic optimality of Hoeffding's likelihood ratio test, which is equivalent to the threshold test of the relative entropy between the empirical distribution and the nominal distribution. The new proof offers an intuitive interpretation and naturally extends to the two-sample test where we show that a similar form of Hoeffding's test, namely a threshold test of the relative entropy between the two empirical distributions is also asymptotically optimal. A strong converse for the two-sample test is also obtained.
Paper Structure (13 sections, 6 theorems, 47 equations, 2 figures)

This paper contains 13 sections, 6 theorems, 47 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Notation and Preliminaries
  4. One-Sample Problem
  5. Two-Sample Problem
  6. Results
  7. One Sample Test
  8. Two Sample Test
  9. Proofs
  10. Proof of One Sample Achievability
  11. Proof of Two-Sample Achievability
  12. Proof of Strong Converse
  13. Conclusion

Key Result

Theorem 1

Let $X^n \sim P$ for $P \in \mathcal{P}^d$. Let $E \subset \mathcal{P}^d$. Then where $P^* = \mathop{\mathrm{arg\,min}}\limits_{F \in E} D\left( F \| P \right)$.

Figures (2)

  • Figure 1: One-Sample Problem
  • Figure 2: Two-Sample Problem

Theorems & Definitions (9)

  • Theorem 1: Sanov's Theorem
  • Lemma 1
  • Theorem 2: Chernoff-Stein Lemma
  • Theorem 3
  • Theorem 4: Achievability
  • Theorem 5: Strong Converse
  • proof
  • proof
  • proof