Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonparametric Inference on Unlabeled Histograms

Yun Ma, Pengkun Yang

TL;DR

This work introduces a nonparametric framework for inferring unlabeled histograms by modeling the multiset of frequency counts as a Poisson mixture governed by a mixing distribution π. The Poisson-NPMLE hat{π} provides a convex optimization-based estimator with strong asymptotic and non-asymptotic guarantees, and supports flexible plug-in estimators for symmetric functionals such as entropy, unseen-species counts, and Rényi measures. A localized NPMLE and bias-correction scheme enable minimax-optimal estimation in large-alphabet regimes, while a penalized variant handles unknown support sizes. Extensive simulations, real-data experiments, and large-language-model evaluations demonstrate improved accuracy, robustness, and scalability in entropy and unseen-element estimation, with practical implications for vocabulary analysis, neuroscience, and AI model evaluation.

Abstract

Statistical inference on histograms and frequency counts plays a central role in categorical data analysis. Moving beyond classical methods that directly analyze labeled frequencies, we introduce a framework that models the multiset of unlabeled histograms via a mixture distribution to better capture unseen domain elements in large-alphabet regime. We study the nonparametric maximum likelihood estimator (NPMLE) under this framework, and establish its optimal convergence rate under the Poisson setting. The NPMLE also immediately yields flexible and efficient plug-in estimators for functional estimation problems, where a localized variant further achieves the optimal sample complexity for a wide range of symmetric functionals. Extensive experiments on synthetic, real-world datasets, and large language models highlight the practical benefits of the proposed method.

Nonparametric Inference on Unlabeled Histograms

TL;DR

This work introduces a nonparametric framework for inferring unlabeled histograms by modeling the multiset of frequency counts as a Poisson mixture governed by a mixing distribution π. The Poisson-NPMLE hat{π} provides a convex optimization-based estimator with strong asymptotic and non-asymptotic guarantees, and supports flexible plug-in estimators for symmetric functionals such as entropy, unseen-species counts, and Rényi measures. A localized NPMLE and bias-correction scheme enable minimax-optimal estimation in large-alphabet regimes, while a penalized variant handles unknown support sizes. Extensive simulations, real-data experiments, and large-language-model evaluations demonstrate improved accuracy, robustness, and scalability in entropy and unseen-element estimation, with practical implications for vocabulary analysis, neuroscience, and AI model evaluation.

Abstract

Statistical inference on histograms and frequency counts plays a central role in categorical data analysis. Moving beyond classical methods that directly analyze labeled frequencies, we introduce a framework that models the multiset of unlabeled histograms via a mixture distribution to better capture unseen domain elements in large-alphabet regime. We study the nonparametric maximum likelihood estimator (NPMLE) under this framework, and establish its optimal convergence rate under the Poisson setting. The NPMLE also immediately yields flexible and efficient plug-in estimators for functional estimation problems, where a localized variant further achieves the optimal sample complexity for a wide range of symmetric functionals. Extensive experiments on synthetic, real-world datasets, and large language models highlight the practical benefits of the proposed method.

Paper Structure

This paper contains 60 sections, 25 theorems, 195 equations, 12 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Model and methodology
  3. NPMLE under the P-model
  4. Applications to functional estimation
  5. Related work
  6. Functional estimation
  7. Plug-in and non-plug-in estimators
  8. Nonparametric maximum likelihood
  9. Notation
  10. The Poisson regime
  11. Counting with Poisson processes
  12. Basic properties of the Poisson NPMLE
  13. Existence and uniqueness
  14. Optimality conditions
  15. Statistical properties
  16. ...and 45 more sections

Key Result

Proposition 1

Let $\hat{p}_i\triangleq N_i/n$. The solution $\hat{\pi}$ in eq:pois-npmle-def exists uniquely and is a discrete distribution with support size no more than the number of distinct elements in $\{N_i\}_{i=1}^k$. In addition, $\hat{\pi}$ is supported on $[1 \wedge \min_{i\in [k]} \hat{p}_i, 1 \wedge \

Figures (12)

  • Figure 1: Histogram and the NPMLE-fitted model.
  • Figure 2: CDFs of the underlying distribution $\pi_P=\frac{1}{10}(4\delta_{\frac{1}{24}}+3\delta_{\frac{1}{12}}+2\delta_{\frac{1}{6}}+\delta_{\frac{1}{4}})$ with $k=10$ and the NPMLE fitted with $n=500, 2000, 5000$. The figure illustrates that the NPMLE assigns nearly the same probability mass as $\pi_P$ around each of its atom.
  • Figure 3: The scaled $\mathsf{KL}$ divergence $k' \cdot \mathsf{KL}(\pi_{N'}\| f_{\hat{\pi}_{k'}})$ under the uniform distribution with true support size $k^\star = 500$ and varying sample sizes $n$. Each curve starts at the number of observed non-zero counts $k$, and the vertical colored line indicates the selected $\hat{k}$ value.
  • Figure 4: Shannon entropy estimation: Panels (a)–(c) plot the RMSE of the large-sample regime, while panels (d)–(f) show the results of the large-alphabet regime.
  • Figure 5: Performance of the penalized NPMLE.
  • ...and 7 more figures

Theorems & Definitions (47)

  • Proposition 1
  • Definition 1: $r$-separation
  • Proposition 2
  • Remark 1
  • Proposition 3
  • Proposition 4
  • Theorem 1
  • Theorem 2
  • Remark 2
  • Theorem 3
  • ...and 37 more