Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimation of discrete distributions with high probability under $χ^2$-divergence

Sirine Louati

TL;DR

The paper studies high-probability estimation of a discrete distribution under $\chi^2$-divergence using i.i.d. samples. It analyzes the classical Laplace (add-one) estimator, deriving sharp upper and lower bounds, and introduces a confidence-dependent smoothing scheme to improve rates. The authors show that the Laplace estimator is minimax optimal among confidence-independent estimators, and they characterize minimax optimal rates with and without confidence calibration, revealing an intrinsic overhead in non-asymptotic guarantees. These results yield practical smoothing strategies and tighten theoretical understanding of divergence-based estimation in high dimensions.

Abstract

We investigate the high-probability estimation of discrete distributions from an \iid sample under $χ^2$-divergence loss. Although the minimax risk in expectation is well understood, its high-probability counterpart remains largely unexplored. We provide sharp upper and lower bounds for the classical Laplace estimator, showing that it achieves optimal performance among estimators that do not rely on the confidence level. We further characterize the minimax high-probability risk for any estimator and demonstrate that it can be attained through a simple smoothing strategy. Our analysis highlights an intrinsic separation between asymptotic and non-asymptotic guarantees, with the latter suffering from an unavoidable overhead. This work sharpens existing guarantees and advances the theoretical understanding of divergence-based estimation.

Estimation of discrete distributions with high probability under $χ^2$-divergence

TL;DR

The paper studies high-probability estimation of a discrete distribution under -divergence using i.i.d. samples. It analyzes the classical Laplace (add-one) estimator, deriving sharp upper and lower bounds, and introduces a confidence-dependent smoothing scheme to improve rates. The authors show that the Laplace estimator is minimax optimal among confidence-independent estimators, and they characterize minimax optimal rates with and without confidence calibration, revealing an intrinsic overhead in non-asymptotic guarantees. These results yield practical smoothing strategies and tighten theoretical understanding of divergence-based estimation in high dimensions.

Abstract

We investigate the high-probability estimation of discrete distributions from an \iid sample under -divergence loss. Although the minimax risk in expectation is well understood, its high-probability counterpart remains largely unexplored. We provide sharp upper and lower bounds for the classical Laplace estimator, showing that it achieves optimal performance among estimators that do not rely on the confidence level. We further characterize the minimax high-probability risk for any estimator and demonstrate that it can be attained through a simple smoothing strategy. Our analysis highlights an intrinsic separation between asymptotic and non-asymptotic guarantees, with the latter suffering from an unavoidable overhead. This work sharpens existing guarantees and advances the theoretical understanding of divergence-based estimation.

Paper Structure

This paper contains 22 sections, 16 theorems, 92 equations, 1 table.

Table of Contents

  1. Introduction
  2. Problem setting
  3. Main questions.
  4. Related Work
  5. About Total Variation, Hellinger and Kullback-Leibler divergences.
  6. The case of the $\chi^2$-divergence.
  7. State-of-the-art bounds and open problems.
  8. Our contribution.
  9. Summary of known bounds.
  10. Notation
  11. Main Results
  12. Optimal guarantees for the Laplace estimator
  13. Upper bound for the Laplace estimator
  14. Lower bound for confidence-independent estimators
  15. Minimax-optimal guarantees for confidence-dependent estimators
  16. ...and 7 more sections

Key Result

Theorem 1

For any $n \geqslant 12, d \geqslant 2$ and $P \in \Delta_d$, the Laplace estimator $\widehat{P}_n$ defined by eq:def-laplace achieves the following guarantee: for any $\delta \in (e^{-n/6}, e^{-2})$, letting $C_1=25900$, we have that

Theorems & Definitions (33)

  • Theorem 1
  • Remark 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • proof : Proof of Lemma \ref{['lem:lower-minimax-tail']}
  • Lemma 2
  • proof : Proof of Lemma \ref{['lem:lower-bound-conf-indep-tail']}
  • proof : Proof of Theorem \ref{['thm:lower-bound-conf-indep-tail']}
  • ...and 23 more