Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence intervals for maximum unseen probabilities, with application to sequential sampling design

Alessandro Colombi, Mario Beraha, Amichai Painsky, Stefano Favaro

Abstract

Discovery problems often require deciding whether additional sampling is needed to detect all categories whose prevalence exceeds a prespecified threshold. We study this question under a Bernoulli product (incidence) model, where categories are observed only through presence--absence across sampling units. Our inferential target is the \emph{maximum unseen probability}, the largest prevalence among categories not yet observed. We develop nonasymptotic, distribution-free upper confidence bounds for this quantity in two regimes: bounded alphabets (finite and known number of categories) and unbounded alphabets (countably infinite under a mild summability condition). We characterise the limits of data-independent worst-case bounds, showing that in the unbounded regime no nontrivial data-independent procedure can be uniformly valid. We then propose data-dependent bounds in both regimes and establish matching lower bounds demonstrating their near-optimality. We compare empirically the resulting procedures in both simulated and real datasets. Finally, we use these bounds to construct sequential stopping rules with finite-sample guarantees, and demonstrate robustness to contamination that introduces spurious low-prevalence categories.

Confidence intervals for maximum unseen probabilities, with application to sequential sampling design

Abstract

Discovery problems often require deciding whether additional sampling is needed to detect all categories whose prevalence exceeds a prespecified threshold. We study this question under a Bernoulli product (incidence) model, where categories are observed only through presence--absence across sampling units. Our inferential target is the \emph{maximum unseen probability}, the largest prevalence among categories not yet observed. We develop nonasymptotic, distribution-free upper confidence bounds for this quantity in two regimes: bounded alphabets (finite and known number of categories) and unbounded alphabets (countably infinite under a mild summability condition). We characterise the limits of data-independent worst-case bounds, showing that in the unbounded regime no nontrivial data-independent procedure can be uniformly valid. We then propose data-dependent bounds in both regimes and establish matching lower bounds demonstrating their near-optimality. We compare empirically the resulting procedures in both simulated and real datasets. Finally, we use these bounds to construct sequential stopping rules with finite-sample guarantees, and demonstrate robustness to contamination that introduces spurious low-prevalence categories.
Paper Structure (24 sections, 11 theorems, 101 equations, 13 figures, 1 table)

This paper contains 24 sections, 11 theorems, 101 equations, 13 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and motivation
  3. Preview of our contributions
  4. Related work
  5. Other applications
  6. Bounded alphabet sizes
  7. A worst-case, data independent, bound
  8. Data-dependent bounds
  9. A matching lower bound
  10. Unbounded alphabet sizes
  11. Impossibility of worst-case, data-independent bounds
  12. A tight, data-dependent, $r$-norm bound
  13. Numerical illustrations
  14. Simulated data with bounded and unbounded alphabets
  15. Bounded vs unbounded bound: when to choose one over the other?
  16. ...and 9 more sections

Key Result

Theorem 2.1

Let $N$ be a sample of size $n$ from the Bernoulli product model with alphabet size $M$. Let $M_{\max}(N)$ be as in eq:mmax_def, fix $\alpha \in (0,1)$ and assume that $\log M=o(n)$. Define Then, for every $p=(p_1,\ldots,p_M)$,

Figures (13)

  • Figure 1: Confidence interval length under Zipf-like Bernoulli probabilities. First row: the sample size is fixed while the alphabet size $M$ varies. Second row: the alphabet size is fixed while the sample size $n$ increases. Each column corresponds to a different value of $\gamma$: $\gamma = 0.25$ (left), $\gamma = 0.5$ (middle), and $\gamma = 1.02$ (right). The y-axis is scaled by a factor of $10^3$ for readability.
  • Figure 2: Confidence interval length for Zipf-like (left), geometric-like (center), and homogeneous (right) Bernoulli probabilities. The vertical dotted line indicates the heuristic threshold in Equation \ref{['eqn:threshold_S']}. The y-axis is rescaled by a factor of $10^3$.
  • Figure 3: Accumulation curves under Zipf-like Bernoulli probabilities. Panels correspond to $\gamma = 1.05$ (left), $\gamma = 0.85$ (center), and $\gamma = 0.75$ (right). Each curve shows the number of distinct symbols observed as a function of the sample size $n$.
  • Figure 4: Length of confidence intervals for four selected TCGA cancer types as a function of the sample size $n$. Panels correspond to BRCA, LUSC, SKCM, and ESCA, from left to right. The dashed and dotted grey horizontal lines refer to probability levels $0.05$ and $0.01$, respectively.
  • Figure 5: Stopping time $N_{\mathrm{stop}}$ (top row) and propotion of missed relevant species (bottom row) for the simulation in Section \ref{['sec:SS4stopping']}.
  • ...and 8 more figures

Theorems & Definitions (18)

  • Theorem 2.1
  • Theorem 2.2
  • Corollary 1
  • Theorem 2.3
  • Proposition 1
  • Proposition 2
  • Theorem 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Theorem 3.4
  • ...and 8 more