ST-BCP: Tightening Coverage Bound for Backward Conformal Prediction via Non-Conformity Score Transformation

TL;DR

ST-BCP tackles the gap between empirical coverage and the bound in Backward Conformal Prediction by learning a data-dependent, exchangeability-preserving transformation of non-conformity scores. The approach derives an optimal monotone (effectively step-like) transformation, introduces an efficient implementation via the $oldsymbol{I}_w$ operator, and proves consistency of the LOO-based bound estimator under mild conditions. Empirically, ST-BCP substantially reduces the coverage gap across CIFAR-10/100 and Tiny-ImageNet while maintaining or improving estimation stability and asymptotic efficiency, and it remains effective under various size-constraint rules and model families. The work advances uncertainty quantification for bounded-prediction sets, offering a scalable, theory-backed method with broad applicability to classification under strict size controls.

Abstract

Conformal Prediction (CP) provides a statistical framework for uncertainty quantification that constructs prediction sets with coverage guarantees. While CP yields uncontrolled prediction set sizes, Backward Conformal Prediction (BCP) inverts this paradigm by enforcing a predefined upper bound on set size and estimating the resulting coverage guarantee. However, the looseness induced by Markov's inequality within the BCP framework causes a significant gap between the estimated coverage bound and the empirical coverage. In this work, we introduce ST-BCP, a novel method that introduces a data-dependent transformation of nonconformity scores to narrow the coverage gap. In particular, we develop a computable transformation and prove that it outperforms the baseline identity transformation. Extensive experiments demonstrate the effectiveness of our method, reducing the average coverage gap from 4.20\% to 1.12\% on common benchmarks.

Figures (6)

• Figure 1: Performance comparison between BCP ($h=s$) and our method ST-BCP ($h=\mathbb{I}_w$) under different datasets. We present kernel density estimation (KDE) plots of the LOO estimator, the empirical miscoverage, and the empirical expectation miscoverage level. All results are obtained using a ResNet50 model under the specified size constraint rule with a calibration set size $n=200$. Note: Distributions that are more concentrated and closer to the empirical coverage indicate superior performance.
• Figure 2: Comparison of LOO estimator convergence rate and stability between BCP and ST-BCP across different calibration set sizes $n$. We report the MSE and STD that are obtained using a ResNet50 model on the CIFAR-10 dataset under the size constraint rule $\mathcal{T}=2$. As $n$ increases, our method ST-BCP ($h=\mathbb{I}_w$) consistently outperforms the baseline BCP ($h=s$).
• Figure 3: Performance comparison between BCP($h=s$) and ST-BCP($h=\mathbb{I}_w$) under different models. We report the MSE and GAP that are obtained on the CIFAR-10 dataset with the calibration set size of $n=200$ under the size constraint rule $\mathcal{T}=2$. Among these models, our method ST-BCP($h=\mathbb{I}_w$) has always outperformed the baseline BCP($h=s$).
• Figure 4: Under different datasets, the MisCov, kernel density estimation plots of the empirical distribution of the LOO estimator, and the empirical miscoverage levels of both are presented. The results were obtained with the size constraint rule, using the ResNet50 model and the calibration set size $n=200$.
• Figure 5: Under different datasets, the MisCov, kernel density estimation plots of the empirical distribution of the LOO estimator, and the empirical miscoverage levels of both are presented. The results were obtained with the size constraint rule, using the ResNet50 model and the calibration set size $n=200$.

Theorems & Definitions (11)

• Theorem 3.1
• Theorem 3.2
• Remark 3.3
• Corollary 3.4
• Theorem 3.5
• Corollary 3.6
• proof
• proof
• proof
• Proposition 8.1