Conformal novelty detection with false discovery rate control at the boundary

TL;DR

This work analyzes boundary control in conformal novelty detection and shows that extending the SL correction can fail to bound the boundary false discovery rate (bFDR) under dependence. It introduces the boundary-safe conformal procedures SLC and ASLC, with variants that leverage subsampling to boost power when calibration data are limited, and proves provable bFDR control at level $\pi_0 \alpha$ or $\alpha$ under suitable assumptions. An adaptive version using Storey style pi0 estimation further improves power while maintaining guarantees, and a monotonicity-based interpretation links bFDR control to conventional FDR control and an empirical Bayes lfdr threshold via isotonic regression (Grenander estimator). Empirical results on simulated data and CIFAR 10 demonstrate that the proposed methods achieve reliable boundary control with competitive power, offering practical tools for reliable conformal novelty detection.

Abstract

Conformal novelty detection is a classical machine learning task for which uncertainty quantification is essential for providing reliable results. Recent work has shown that the BH procedure applied to conformal p-values controls the false discovery rate (FDR). Unfortunately, the BH procedure can lead to over-optimistic assessments near the rejection threshold, with an increase of false discoveries at the margin as pointed out by Soloff et al. (2024). This issue is solved therein by the support line (SL) correction, which is proven to control the boundary false discovery rate (bFDR) in the independent, non-conformal setting. The present work extends the SL method to the conformal setting: first, we show that the SL procedure can violate the bFDR control in this specific setting. Second, we propose several alternatives that provably control the bFDR in the conformal setting. Finally, numerical experiments with both synthetic and real data support our theoretical findings and show the relevance of the new proposed procedures.

Figures (17)

• Figure 1: Illustration of FDR controlling issue in the conformal novelty detection task. Left: pictures around BH's rejection threshold. Right: distribution of FDP in several rejection sets. Detected animal images are false discoveries. $\alpha = 0.2$, $m_0/m = 0.5$.
• Figure 2: Left: Greatest Convex Minorant (GCM) of $p$-value order statistics (black) and shifted sorted $p$-values $\tilde{p}_{(k)}\coloneqq p_{\sigma(k)}+\frac{k}{n+1}$ (red). Right: Least Concave Majorant (LCM) of $F_m$ (black) and $\tilde{F}_m$ (red). SL makes 11 rejections, while SLC makes 6 rejections, as determined by the points of tangency between the line of slope $\alpha$ (or slope $1/\alpha$) and the GCM (or LCM) of the $p$-values. Here, $m=32,n=64,m_0=16,\alpha=0.8$, $f_1= \text{Uniform}(0.5,1.5)$, and $f_0=\text{Uniform}(0,1)$.
• Figure 3: Comparison of SL, SLC, SLC+ applied to simulated conformal p-values.
• Figure 4: Comparison of SLC, SLC+, and their adaptive variants ASLC, ASLC+ applied to simulated conformal p-values.
• Figure 5: Comparison of BH and SL variants applied to conformal p-values derived from CIFAR 10 (bFDR level $0.2$). BH at level $0.1$ and $0.2$ both consistently yield larger rejection sets compared to SL variants (panel (a)). The FDP among the data points rejected by BH (level $0.2$) but not by SLC+ (in coral) is much higher than that of BH (level $0.2$) (in gold), which centers around $\pi_0 \alpha = 0.1$ (panel (b)).

Theorems & Definitions (32)

• Proposition 1
• Remark 2.1
• Remark 2.2
• Theorem 1
• Definition 1
• Corollary 1
• Theorem 2
• Definition 2
• Theorem 3
• Definition 3