Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors

Oliver Hennhöfer, Christine Preisach

TL;DR

This work formalizes leave-one-out-, bootstrap-, and cross-conformal anomaly detectors by adapting conformal prediction to unsupervised one-class classifiers, enabling valid p-values under exchangeability and mitigating calibration-data limitations. It shows that resampling-conformal methods expand calibration information and can yield higher statistical power while controlling the batch-wise FDR at level $\alpha$ via the Benjamini-Hochberg procedure, even in low-data regimes. Through two large-scale experiments on ten benchmark datasets with three base detectors, the authors demonstrate that Jackknife-, CV-, and bootstrap-based conformal anomaly detectors offer reliable FDR control and improved power relative to split-conformal, with calibration-set size effects tapering in high-data regimes. The methods are model-agnostic, integrate with common anomaly detectors, and are accompanied by public software for reproducibility, facilitating practical uncertainty quantification in real-world anomaly detection systems.

Abstract

The requirement of uncertainty quantification for anomaly detection systems has become increasingly important. In this context, effectively controlling Type I error rates ($α$) without compromising the statistical power ($1-β$) of these systems can build trust and reduce costs related to false discoveries. The field of conformal anomaly detection emerges as a promising approach for providing respective statistical guarantees by model calibration. However, the dependency on calibration data poses practical limitations - especially within low-data regimes. In this work, we formally define and evaluate leave-one-out-, bootstrap-, and cross-conformal methods for anomaly detection, incrementing on methods from the field of conformal prediction. Looking beyond the classical inductive conformal anomaly detection, we demonstrate that derived methods for calculating resampling-conformal $p$-values strike a practical compromise between statistical efficiency (full-conformal) and computational efficiency (split-conformal) as they make more efficient use of available data. We validate derived methods and quantify their improvements for a range of one-class classifiers and datasets.

Leave-One-Out-, Bootstrap- and Cross-Conformal Anomaly Detectors

TL;DR

This work formalizes leave-one-out-, bootstrap-, and cross-conformal anomaly detectors by adapting conformal prediction to unsupervised one-class classifiers, enabling valid p-values under exchangeability and mitigating calibration-data limitations. It shows that resampling-conformal methods expand calibration information and can yield higher statistical power while controlling the batch-wise FDR at level via the Benjamini-Hochberg procedure, even in low-data regimes. Through two large-scale experiments on ten benchmark datasets with three base detectors, the authors demonstrate that Jackknife-, CV-, and bootstrap-based conformal anomaly detectors offer reliable FDR control and improved power relative to split-conformal, with calibration-set size effects tapering in high-data regimes. The methods are model-agnostic, integrate with common anomaly detectors, and are accompanied by public software for reproducibility, facilitating practical uncertainty quantification in real-world anomaly detection systems.

Abstract

The requirement of uncertainty quantification for anomaly detection systems has become increasingly important. In this context, effectively controlling Type I error rates () without compromising the statistical power () of these systems can build trust and reduce costs related to false discoveries. The field of conformal anomaly detection emerges as a promising approach for providing respective statistical guarantees by model calibration. However, the dependency on calibration data poses practical limitations - especially within low-data regimes. In this work, we formally define and evaluate leave-one-out-, bootstrap-, and cross-conformal methods for anomaly detection, incrementing on methods from the field of conformal prediction. Looking beyond the classical inductive conformal anomaly detection, we demonstrate that derived methods for calculating resampling-conformal -values strike a practical compromise between statistical efficiency (full-conformal) and computational efficiency (split-conformal) as they make more efficient use of available data. We validate derived methods and quantify their improvements for a range of one-class classifiers and datasets.
Paper Structure (13 sections, 4 theorems, 18 equations, 1 figure, 5 tables, 1 algorithm)

This paper contains 13 sections, 4 theorems, 18 equations, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Leave-one-out-, Bootstrap- and Cross-Conformal Anomaly Detection
  5. $\text{Jackknife}_{\text{AD}}$ and $\text{Jackknife+}_{\text{AD}}$
  6. $\text{CV}_{\text{AD}}$ and $\text{CV+}_{\text{AD}}$
  7. $\text{Jackknife- and Jackknife+-after-Bootstrap}_{AD}$
  8. Multiple Testing, PRDS, and the Benjamini-Hochberg Procedure
  9. Evaluation
  10. Setup
  11. Implementation Details
  12. Results and Discussion
  13. Conclusion

Key Result

Proposition 3.3

If the inliers in $\mathcal{D}_{\text{calib}}$ are exchangable with themselves and with $X_{n+1}$, then $\mathbb{P}_{\mathcal{H}_{0}}[\hat{u}(X_{n+1}) \le \alpha] \le \alpha.$ for all $\alpha \in (0,1)$.

Figures (1)

  • Figure 1: Non-exhaustive taxonomy of the field of conformal inference with conformal prediction, conformal anomaly detection, and the derived family of resampling-conformal methods for anomaly detection.

Theorems & Definitions (7)

  • Definition 3.1
  • Definition 3.2: Super-Uniformity
  • Proposition 3.3: e.g. from Bates2023
  • Proposition 4.1: cf. Liang2024
  • Definition 5.1: PRDS, e.g., from Benjamini2001
  • Theorem 5.2: cf. Bates2023
  • Theorem 5.3: e.g., from Benjamini2001