Estimating the Shannon Entropy Using the Pitman--Yor Process

TL;DR

This work tackles estimating the Shannon entropy $H= -\sum_i p_i\log p_i$ when the true support $K$ is unknown and potentially large. It introduces a Pitman--Yor process–based estimator via a Dirichlet--Pitman--Yor mixture model to account for unseen species, enabling consistent entropy estimation in small-sample, high-dimensional settings. The methodology derives the MPY predictive distribution, develops data-driven hyperparameter selection using cross-entropy upper bounds and Good--Turing estimates, and proves consistency under regularly varying tails; it is complemented by simulations and real-data applications (tropical trees and 20 Newsgroups) showing improved bias correction and robustness. The approach provides a principled way to quantify diversity when $K$ is unknown and demonstrates practical impact for ecology and text analytics in challenging sampling regimes.

Abstract

The Shannon entropy is a fundamental measure for quantifying diversity and model complexity in fields such as information theory, ecology, and genetics. However, many existing studies assume that the number of species is known, an assumption that is often unrealistic in practice. In recent years, efforts have been made to relax this restriction. Motivated by these developments, this study proposes an entropy estimation method based on the Pitman--Yor process, a representative approach in Bayesian nonparametrics. By approximating the true distribution as an infinite-dimensional process, the proposed method enables stable estimation even when the number of observed species is smaller than the true number of species. This approach provides a principled way to deal with the uncertainty in species diversity and enhances the reliability and robustness of entropy-based diversity assessment. In addition, we investigate the convergence property of the Shannon entropy for regularly varying distributions and use this result to establish the consistency of the proposed estimator. Finally, we demonstrate the effectiveness of the proposed method through numerical experiments.

Figures (5)

• Figure 1: The probability mass functions of the marginal Pitman--Yor process. The upper-left and upper-right panels correspond to $\alpha=49$ and $\alpha=99$, respectively. The top row shows probability mass functions for different values of the discount parameter $d$: $d=0$ (gray, geometric), $d=0.5$ (light blue), and $d=0.75$ (dark blue). In the bottom row, the parameters $(d,\alpha)$ are chosen such that $P_{d,\alpha}^{\mathrm{MPY}}(1)=0.02$ in all cases.
• Figure 2: Behavior of the Kullback--Leibler divergence $D_{KL}(\bm{p}||\bm{q})$ and the difference between the cross-entropy upper bound and the true entropy, $f(d,\alpha)-H(\bm{p})$, as functions of $\alpha \in [0,1000)$. The discount parameter is fixed at $d=0$ (left column) and $d=0.5$ (right column). In the top row, the true probability vector $\bm{p}$ is drawn from $\mathrm{Dir}_{500}(0.1)$, while in the bottom row $\bm{p}$ is drawn from $\mathrm{Dir}_{500}(10)$.
• Figure 3: Comparing the four different entropy estimators (MLE, Miller--Madow, Chao--Shen and the proposed estimator) in five different sampling scenarios. The estimators are compared in terms of MSE of the estimated entropy. The dimension is fixed at $K = 5000$ while the sample size $N$ varies from 10 to 20000.
• Figure 4: Estimates of the Shannon entropy for each plot in the BCI dataset
• Figure 5: Estimates of the Shannon entropy for each document in the 20 Newsgroups dataset

Theorems & Definitions (40)

• Remark 1
• Proposition 2.1
• proof
• Remark 2
• Proposition 2.2
• proof
• Definition 3.1
• Remark 3
• Remark 4
• Proposition 3.2