The Unseen Species Problem Revisited
Edward Eriksson
TL;DR
The paper rigorously analyzes the unseen species problem and its generalizations, introducing the generalized unseen-species framework with bounded-arity incidence data. It shows the classical Good-Toulmin estimator is minimax-optimal up to a factor of $1+r$ in the near-future regime, and develops a data-driven linear-estimator optimization (via $G_H$) for the intermediate regime, yielding substantial improvements over Smoothed Good-Toulmin in worst-case and practical settings. For the distant-future regime, the authors leverage power-law tail modeling and a ratio-$\alpha$ estimator to achieve favorable Gaussian limits and extend these results to generalized, bounded-arity data with concentration analyses based on size-biased couplings. Across all regimes, the work provides principled uncertainty quantification through CLTs and variance proxies, plus extensive experiments on real and synthetic data demonstrating improved performance in datasets rich with rare species and incidence structures.
Abstract
The unseen species problem is a classical problem in statistics. It asks us to, given n i.i.d. samples from an unknown discrete distribution over an unknown set, predict how many never before seen outcomes would be observed if m additional samples were collected. For small m we show the classical but poorly understood Good-Toulmin estimator to be minimax optimal to within a factor 2 and resolve the open problem of constructing principled prediction intervals for it. For intermediate m we propose a new estimator which achieves the minimax error for linear estimators up to an explicit multiplicative constant. Our estimator vastly outperforms the standard Smoothed Good-Toulmin estimator in the worst case and performs substantially better on several real data sets, namely those with many rare species. For large m we show that a previously mentioned estimator which did not have known rate guarantees actually achieves a marginally better rate than subsequent work. We find that this marginal rate improvement translates to meaningfully better performance in practice. We show in all three regimes that the same methods also achieve the same rate on incidence data, without further independence assumptions, provided that the sets are of bounded size. We establish, by means of bounded size biased couplings, concentration for some natural functionals of sequences of i.i.d. discrete-set-valued random variables which may be of independent interest.