Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty Quantification and Confidence Intervals for Naive Rare-Event Estimators

Yuanlu Bai, Henry Lam

TL;DR

This work analyzes uncertainty quantification for estimating rare-event probabilities $p$ from naive Bernoulli data using $\hat{p}$. It evaluates the validity and tightness of standard CIs (CLT, Wilson) and the Exact CI, revealing potential under-coverage in the rare-event regime and conservativeness of the Exact CI. To address these issues, the authors derive two new valid CIs via Chernoff's inequality and Berry-Esseen bounds, and extend all results to a targeted stopping setting where sampling ends after a fixed number of successes. Numerical experiments illustrate the trade-offs: CLT/Wilson are not always valid but are close in width to the new valid CIs, while Chernoff/BE are conservative but offer valuable insight into interval tightness and finite-sample behavior. The findings guide practitioners on choosing CIs for rare-event estimation, balancing guaranteed coverage with interval sharpness and practical feasibility.

Abstract

We consider the estimation of rare-event probabilities using sample proportions output by naive Monte Carlo or collected data. Unlike using variance reduction techniques, this naive estimator does not have a priori relative efficiency guarantee. On the other hand, due to the recent surge of sophisticated rare-event problems arising in safety evaluations of intelligent systems, efficiency-guaranteed variance reduction may face implementation challenges which, coupled with the availability of computation or data collection power, motivate the use of such a naive estimator. In this paper we study the uncertainty quantification, namely the construction, coverage validity and tightness of confidence intervals, for rare-event probabilities using only sample proportions. In addition to the known normality, Wilson's and exact intervals, we investigate and compare them with two new intervals derived from Chernoff's inequality and the Berry-Esseen theorem. Moreover, we generalize our results to the natural situation where sampling stops by reaching a target number of rare-event hits. Our findings show that the normality and Wilson's intervals are not always valid, but they are close to the newly developed valid intervals in terms of half-width. In contrast, the exact interval is conservative, but safely guarantees the attainment of the nominal confidence level. Our new intervals, while being more conservative than the exact interval, provide useful insights in understanding the tightness of the considered intervals.

Uncertainty Quantification and Confidence Intervals for Naive Rare-Event Estimators

TL;DR

This work analyzes uncertainty quantification for estimating rare-event probabilities from naive Bernoulli data using . It evaluates the validity and tightness of standard CIs (CLT, Wilson) and the Exact CI, revealing potential under-coverage in the rare-event regime and conservativeness of the Exact CI. To address these issues, the authors derive two new valid CIs via Chernoff's inequality and Berry-Esseen bounds, and extend all results to a targeted stopping setting where sampling ends after a fixed number of successes. Numerical experiments illustrate the trade-offs: CLT/Wilson are not always valid but are close in width to the new valid CIs, while Chernoff/BE are conservative but offer valuable insight into interval tightness and finite-sample behavior. The findings guide practitioners on choosing CIs for rare-event estimation, balancing guaranteed coverage with interval sharpness and practical feasibility.

Abstract

We consider the estimation of rare-event probabilities using sample proportions output by naive Monte Carlo or collected data. Unlike using variance reduction techniques, this naive estimator does not have a priori relative efficiency guarantee. On the other hand, due to the recent surge of sophisticated rare-event problems arising in safety evaluations of intelligent systems, efficiency-guaranteed variance reduction may face implementation challenges which, coupled with the availability of computation or data collection power, motivate the use of such a naive estimator. In this paper we study the uncertainty quantification, namely the construction, coverage validity and tightness of confidence intervals, for rare-event probabilities using only sample proportions. In addition to the known normality, Wilson's and exact intervals, we investigate and compare them with two new intervals derived from Chernoff's inequality and the Berry-Esseen theorem. Moreover, we generalize our results to the natural situation where sampling stops by reaching a target number of rare-event hits. Our findings show that the normality and Wilson's intervals are not always valid, but they are close to the newly developed valid intervals in terms of half-width. In contrast, the exact interval is conservative, but safely guarantees the attainment of the nominal confidence level. Our new intervals, while being more conservative than the exact interval, provide useful insights in understanding the tightness of the considered intervals.
Paper Structure (27 sections, 10 theorems, 71 equations, 4 figures, 4 tables)

This paper contains 27 sections, 10 theorems, 71 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Setting and Motivation
  3. Overview of Confidence Intervals
  4. Confidence Intervals under Standard Setting
  5. Confidence Intervals under Targeted Stopping
  6. Summary of Main Results
  7. Developments under Standard Setting
  8. Derivation of New Confidence Intervals
  9. Chernoff's CI.
  10. B-E CI.
  11. Analyses of Half-Widths
  12. CLT CI.
  13. Wilson's CI.
  14. Chernoff's CI.
  15. B-E CI.
  16. ...and 12 more sections

Key Result

Theorem 1

The interval given by is a valid $(1-\alpha)$-level CI for $p$, for any finite sample $n$. That is, $\mathbb{P}(\hat{p}_l^{Chernoff}\leq p\leq \hat{p}_u^{Chernoff})\geq1-\alpha$ for any $n$.

Figures (4)

  • Figure 1: Comparisons of the positions of confidence upper and lower bounds under the standard setting. In this figure, "valid" means that the CI has valid coverage in the sense that the actual coverage probability always reaches the nominal confidence level.
  • Figure 2: Comparisons of the positions of confidence upper and lower bounds under the targeted stopping setting. In this figure, "valid" means that the CI has valid coverage in the sense that the actual coverage probability always reaches the nominal confidence level.
  • Figure 3: Average values of the confidence upper and lower bounds under the standard setting.
  • Figure 4: Average values of the confidence upper and lower bounds under the targeted stopping setting.

Theorems & Definitions (20)

  • Definition 1: CLT CI under Standard Setting
  • Definition 2: Wilson's CI under Standard Setting
  • Definition 3: Exact CI under Standard Setting
  • Definition 4: Chernoff's CI under Standard Setting
  • Definition 5: B-E CI under Standard Setting
  • Definition 6: CLT CI under Targeted Stopping
  • Definition 7: Wilson's CI under Targeted Stopping
  • Definition 8: Exact CI under Targeted Stopping
  • Definition 9: Chernoff's CI under Targeted Stopping
  • Definition 10: B-E CI under Targeted Stopping
  • ...and 10 more