Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distribution Estimation with Side Information

Haricharan Balasundaram, Andrew Thangaraj

TL;DR

This work addresses discrete distribution estimation from i.i.d. samples when side information is available, motivated by large alphabets in language data. It develops two models: a local information model where the true distribution lies in an $\ell_2$-ball around a guess $\pi^{(0)}$ and a partial-ordering model that partitions the alphabet into low- and high-probability sets. The authors propose an interpolation estimator between the empirical distribution and the guess in Model 1 and a two-level Good-Turing estimator in Model 2, establishing minimax upper and lower bounds for the squared $\ell_2$ loss and showing that side information can yield substantial gains, especially for small sample sizes. Simulations on natural language data and synthetic distributions corroborate the theoretical results, highlighting practical benefits for language modeling tasks where semantic side information is natural.

Abstract

We consider the classical problem of discrete distribution estimation using i.i.d. samples in a novel scenario where additional side information is available on the distribution. In large alphabet datasets such as text corpora, such side information arises naturally through word semantics/similarities that can be inferred by closeness of vector word embeddings, for instance. We consider two specific models for side information--a local model where the unknown distribution is in the neighborhood of a known distribution, and a partial ordering model where the alphabet is partitioned into known higher and lower probability sets. In both models, we theoretically characterize the improvement in a suitable squared-error risk because of the available side information. Simulations over natural language and synthetic data illustrate these gains.

Distribution Estimation with Side Information

TL;DR

This work addresses discrete distribution estimation from i.i.d. samples when side information is available, motivated by large alphabets in language data. It develops two models: a local information model where the true distribution lies in an -ball around a guess and a partial-ordering model that partitions the alphabet into low- and high-probability sets. The authors propose an interpolation estimator between the empirical distribution and the guess in Model 1 and a two-level Good-Turing estimator in Model 2, establishing minimax upper and lower bounds for the squared loss and showing that side information can yield substantial gains, especially for small sample sizes. Simulations on natural language data and synthetic distributions corroborate the theoretical results, highlighting practical benefits for language modeling tasks where semantic side information is natural.

Abstract

We consider the classical problem of discrete distribution estimation using i.i.d. samples in a novel scenario where additional side information is available on the distribution. In large alphabet datasets such as text corpora, such side information arises naturally through word semantics/similarities that can be inferred by closeness of vector word embeddings, for instance. We consider two specific models for side information--a local model where the unknown distribution is in the neighborhood of a known distribution, and a partial ordering model where the alphabet is partitioned into known higher and lower probability sets. In both models, we theoretically characterize the improvement in a suitable squared-error risk because of the available side information. Simulations over natural language and synthetic data illustrate these gains.
Paper Structure (18 sections, 13 theorems, 39 equations, 5 figures)

This paper contains 18 sections, 13 theorems, 39 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Notation and Preliminaries
  3. Notation
  4. Preliminaries
  5. Effectiveness of the empirical estimator
  6. Loss restricted to symbols in $S_l$
  7. Problem Formulation and Results
  8. Model 1
  9. Model 2
  10. Simulations
  11. Conclusions
  12. Proofs
  13. Proof of Theorem \ref{['thm:ub']}
  14. Proof of Theorem \ref{['lem:lecam_lb']}
  15. Proof of Theorem \ref{['thm:uniform_lb']}
  16. ...and 3 more sections

Key Result

Theorem 2.1

For a distribution $\pi$ and the empirical estimator $\pi^{(\textsf{em})}$,

Figures (5)

  • Figure 1: Estimation errors vs. number of samples for the Empirical and Interpolation Estimators for $\pi^{(0)}$ from 'dataset' and 'sample'. All error bars are for $10$ independent repetitions.
  • Figure 2: Estimation errors vs Delta.
  • Figure 3: Two-level vs one-level estimate for the two-level distribution for $d = 10000$.
  • Figure 4: Distribution Estimates: Two-level vs One-level.
  • Figure 5: Error in Distribution Estimation: Two-level vs One-level.

Theorems & Definitions (13)

  • Theorem 2.1
  • Theorem 2.2
  • Theorem 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Lemma 3.5
  • Theorem 3.6
  • Lemma 6.1
  • Lemma 6.2
  • ...and 3 more