Gaussian credible intervals in Bayesian nonparametric estimation of the unseen

TL;DR

This work addresses the unseen-species problem under a Bayesian nonparametric framework using the Pitman–Yor prior. It introduces a Gaussian central limit theorem-based method to construct large-$m$ credible intervals for $K_{n,m}$ that are centered on the BNP estimator and do not require Monte Carlo sampling, extending to $oldsymbol{ m\alpha}\in[0,1)$ and $ heta>-oldsymbol{ m\alpha}$. By scaling $(n,j, heta)$ with $m$ and deriving explicit expressions for the limiting mean and variance, the authors obtain analytical Gaussian intervals with strong coverage properties, outperforming Mittag-Leffler-based intervals in finite samples. The approach is validated on synthetic distributions (Zipf, Dirichlet–Multinomial, Uniform) and real EST data, showing improved coverage and competitive interval lengths, with practical implications for planning additional sampling in genomic and ecological studies.

Abstract

The unseen-species problem assumes $n\geq1$ samples from a population of individuals belonging to different species, possibly infinite, and calls for estimating the number $K_{n,m}$ of hitherto unseen species that would be observed if $m\geq1$ new samples were collected from the same population. This is a long-standing problem in statistics, which has gained renewed relevance in biological and physical sciences, particularly in settings with large values of $n$ and $m$. In this paper, we adopt a Bayesian nonparametric approach to the unseen-species problem under the Pitman-Yor prior, and propose a novel methodology to derive large $m$ asymptotic credible intervals for $K_{n,m}$, for any $n\geq1$. By leveraging a Gaussian central limit theorem for the posterior distribution of $K_{n,m}$, our method improves upon competitors in two key aspects: firstly, it enables the full parameterization of the Pitman-Yor prior, including the Dirichlet prior; secondly, it avoids the need of Monte Carlo sampling, enhancing computational efficiency. We validate the proposed method on synthetic and real data, demonstrating that it improves the empirical performance of competitors by significantly narrowing the gap between asymptotic and exact credible intervals for any $m\geq1$.

Figures (8)

• Figure 1: BNP estimates of $K_{n,m}$ (solid line --) with 95% exact credible intervals (dashed line - -), Mittag-Leffler credible intervals (violet) and Gaussian credible intervals (pink), as a function of $m \in [0, 5n]$. Synthetic datasets generated from the following discrete distributions: A) Zipf distribution on $\{0,1,\ldots,300\}$ with parameter $2$, $n=977$, $j=300$, and estimated $(\alpha,\theta)=(0.54,\,26.67)$; B) Zipf distribution on $\{0,1,\ldots,100\}$ with parameter $1.5$, $n=1877$, $j=100$, and estimated $(\alpha,\theta)=(0.38,\,4.66)$; C) Pólya distribution on $\{0,1,\ldots,500\}$ with parameter (2, 2, 500, 500, ..., 500), $n=2,000$, $j=227$, and estimated $(\alpha,\theta)=(0.69,\,1.80)$; D) Uniform distribution on $\{0,1,\ldots,500\}$, with $n=2,000$, $j=447$, and estimated $(\alpha,\theta)=(0,\,178.48)$. The parameter $(\alpha,\theta)$ is estimated through an empirical Bayes procedure Fav(09).
• Figure 2: BNP estimates of $K_{n,m}$ (solid line --) with 95% exact credible intervals (dashed line - -), Mittag-Leffler credible intervals (violet) and Gaussian credible intervals (pink), as a function of $m$, for $m \in [0, 5n]$.
• Figure E.1: Coverage of the Mittag-Leffler credible interval (blue) and of the Gaussian credible interval (red) as a function of $m \in [0, 5n]$.
• Figure :
• Figure :

Theorems & Definitions (17)

• Remark 1
• Theorem 3.1
• Theorem 3.2
• Proposition 1: Berry-Esseen theorem for $Q_m(z)$
• Proposition 2
• Proposition 3
• proof
• Proposition 4
• proof
• Lemma D.1