Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimation of discrete distributions in relative entropy, and the deviations of the missing mass

Jaouad Mourtada

TL;DR

The paper provides sharp non-asymptotic high-probability guarantees for estimating discrete distributions under KL divergence. It first analyzes the Laplace (add-one) estimator, establishing a tight upper bound and proving its optimality among confidence-independent methods, while also deriving a matching lower bound. It then proves minimax-optimal high-probability guarantees for confidence-dependent smoothing and demonstrates that an adaptive, data-driven smoothing approach can beat fixed smoothing in sparse regimes. A major contribution is the introduction of effective sparsity concepts and adaptive estimators that leverage them to achieve near-optimal rates across varying support sizes, together with a sharp high-probability bound on missing mass. The results reveal a fundamental separation between asymptotic rates and uniform non-asymptotic guarantees, and they quantify the statistical-computational trade-offs in high-probability discrete distribution estimation.

Abstract

We study the problem of estimating a distribution over a finite alphabet from an i.i.d. sample, with accuracy measured in relative entropy (Kullback-Leibler divergence). While optimal bounds on the expected risk are known, high-probability guarantees remain less well-understood. First, we analyze the classical Laplace (add-one) estimator, obtaining matching upper and lower bounds on its performance and establishing its optimality among confidence-independent estimators. We then characterize the minimax-optimal high-probability risk and show that it is achieved by a simple confidence-dependent smoothing technique. Notably, the optimal non-asymptotic risk incurs an additional logarithmic factor compared to the ideal asymptotic rate. Next, motivated by regimes in which the alphabet size exceeds the sample size, we investigate methods that adapt to the sparsity of the underlying distribution. We introduce an estimator using data-dependent smoothing, for which we establish a high-probability risk bound depending on two effective sparsity parameters. As part of our analysis, we also derive a sharp high-probability upper bound on the missing mass.

Estimation of discrete distributions in relative entropy, and the deviations of the missing mass

TL;DR

The paper provides sharp non-asymptotic high-probability guarantees for estimating discrete distributions under KL divergence. It first analyzes the Laplace (add-one) estimator, establishing a tight upper bound and proving its optimality among confidence-independent methods, while also deriving a matching lower bound. It then proves minimax-optimal high-probability guarantees for confidence-dependent smoothing and demonstrates that an adaptive, data-driven smoothing approach can beat fixed smoothing in sparse regimes. A major contribution is the introduction of effective sparsity concepts and adaptive estimators that leverage them to achieve near-optimal rates across varying support sizes, together with a sharp high-probability bound on missing mass. The results reveal a fundamental separation between asymptotic rates and uniform non-asymptotic guarantees, and they quantify the statistical-computational trade-offs in high-probability discrete distribution estimation.

Abstract

We study the problem of estimating a distribution over a finite alphabet from an i.i.d. sample, with accuracy measured in relative entropy (Kullback-Leibler divergence). While optimal bounds on the expected risk are known, high-probability guarantees remain less well-understood. First, we analyze the classical Laplace (add-one) estimator, obtaining matching upper and lower bounds on its performance and establishing its optimality among confidence-independent estimators. We then characterize the minimax-optimal high-probability risk and show that it is achieved by a simple confidence-dependent smoothing technique. Notably, the optimal non-asymptotic risk incurs an additional logarithmic factor compared to the ideal asymptotic rate. Next, motivated by regimes in which the alphabet size exceeds the sample size, we investigate methods that adapt to the sparsity of the underlying distribution. We introduce an estimator using data-dependent smoothing, for which we establish a high-probability risk bound depending on two effective sparsity parameters. As part of our analysis, we also derive a sharp high-probability upper bound on the missing mass.
Paper Structure (66 sections, 30 theorems, 232 equations)

This paper contains 66 sections, 30 theorems, 232 equations.

Table of Contents

  1. Introduction
  2. Problem setting
  3. Empirical distribution.
  4. Laplace smoothing.
  5. Main questions.
  6. Existing guarantees
  7. High-probability guarantees for the Laplace estimator.
  8. Upper bound for an alternative estimator.
  9. Paper outline
  10. Related work
  11. High-probability guarantees in relative entropy.
  12. Other aspects of estimation of discrete distributions.
  13. Concentration properties of the empirical distribution.
  14. Missing mass.
  15. Notation
  16. ...and 51 more sections

Key Result

Theorem 1

For any $n \geqslant 12, d \geqslant 2$ and $P \in \mathcal{P}_d$, the Laplace estimator $\widehat{P}_n$ defined by eq:def-laplace achieves the following guarantee: for any $\delta \in (e^{-n/6}, e^{-2})$,

Theorems & Definitions (57)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • Proposition 1
  • Corollary 1
  • Definition 1
  • proof
  • Definition 2
  • ...and 47 more