Cost-Sensitive Conformal Training with Provably Controllable Learning Bounds

TL;DR

This work addresses the inefficiency of traditional conformal prediction (CP) training that relies on differentiable surrogates for the hard indicator of prediction-set size. It introduces Rank-Weighted Cross-Entropy (RWCE), a cost-sensitive objective that weights cross-entropy by the rank of the true label, thereby indirectly minimizing the CP set size without surrogate relaxations. The authors prove that the expected CP set size is upper bounded by the expected true-label rank and provide a generalization bound for RWCE, with empirical results showing substantial reductions in average prediction-set size (around 21% on average across benchmarks) while maintaining valid coverage. The method demonstrates strong, cross-domain performance (vision and NLP), offering a practical and theoretically grounded route to more efficient uncertainty quantification in deployed models.

Abstract

Conformal prediction (CP) is a general framework to quantify the predictive uncertainty of machine learning models that uses a set prediction to include the true label with a valid probability. To align the uncertainty measured by CP, conformal training methods minimize the size of the prediction sets. A typical way is to use a surrogate indicator function, usually Sigmoid or Gaussian error function. However, these surrogate functions do not have a uniform error bound to the indicator function, leading to uncontrollable learning bounds. In this paper, we propose a simple cost-sensitive conformal training algorithm that does not rely on the indicator approximation mechanism. Specifically, we theoretically show that minimizing the expected size of prediction sets is upper bounded by the expected rank of true labels. To this end, we develop a rank weighting strategy that assigns the weight using the rank of true label on each data sample. Our analysis provably demonstrates the tightness between the proposed weighted objective and the expected size of conformal prediction sets. Extensive experiments verify the validity of our theoretical insights, and superior empirical performance over other conformal training in terms of predictive efficiency with 21.38% reduction for average prediction set size.

Figures (5)

• Figure 1: Large approximation error of the learning objective (soft set size) from the true objective (hard set size) during training of ConfTr stutz2021learning on CIFAR-100 using ResNet (a) and DenseNet (b). The learning objective is the differentiable measure of prediction set sizes by smooth approximations of the indicator function (e.g., Sigmoid), while the hard set size reflects actual true objectives based on the indicator function. The large and persistent gap between the two highlights the loose approximation introduced by approximated functions, supporting our motivation to avoid indicator approximations and directly optimize a tight upper bound on the true prediction set size.
• Figure 2: Justification experiments using ResNet (Top row) and DenseNet (Bottom row) on CIFAR-100. (a) the training loss convergence of RWCE. The results demonstrate that RWCE converges smoothly and stably on both architectures. (b) the RWCE training loss inflated by $1$ (according to remark \ref{['remark:new_objective_bounds_rank']}) with the actual APSS on the validation set. The loss closely upper bounds APSS with a small and stable gap, indicating that RWCE effectively approximates and directly minimizes the true set size objective. (c) the APSS of RWCE, ConfTr, and CUT calibrated by HPS score. RWCE consistently achieves smaller prediction sets on both architectures, confirming its superior efficiency in directly minimizing set size.
• Figure 3: Empirical validation of the alignment assumption in Theorem \ref{['theorem:new_objective_upper_bounds_expected_rank']}, which states that $\mathbb{E}[(R(X,Y) - 1)(\ell(X,Y) - 1)] \geq -\mathbb{E}[\ell(X,Y)]$ during training on CIFAR-100 using ResNet (a) and DenseNet (b). We plot the left-hand side (expected rank–loss interaction term, blue) and right-hand side (negative expected cross-entropy loss, orange). In both cases, the blue curve remains significantly above the orange curve throughout training, confirming that the inequality holds in practice.
• Figure 4: Justification experiments using ResNet (Top row) and DenseNet (Bottom row) on Caltech-101. (a) the training loss convergence of RWCE . The results demonstrate that RWCE converges smoothly and stably on both architectures. (b) the RWCE training loss inflated with $1$ (according to remark \ref{['remark:new_objective_bounds_rank']}) with the actual APSS on the validation set. The loss closely upper bounds APSS with a small and stable gap, indicating that RWCE effectively approximates and directly minimizes the true set size objective. (c) the APSS of RWCE, ConfTr, and CUT calibrated by HPS score. RWCE consistently achieves smaller prediction sets on both architectures, confirming its superior efficiency in directly minimizing set size.
• Figure 5: Empirical validation of the alignment assumption in Theorem \ref{['theorem:new_objective_upper_bounds_expected_rank']}, which states that $\mathbb{E}[(R(X,Y) - 1)(\ell(X,Y) - 1)] \geq -\mathbb{E}[\ell(X,Y)]$ during training on Caltech-101 using ResNet (a) and DenseNet (b). We plot the left-hand side (expected rank–loss interaction term, blue) and right-hand side (negative expected cross-entropy loss, orange). In both cases, the blue curve remains significantly above the orange curve throughout training, confirming that the inequality holds in practice.

Theorems & Definitions (18)

• Theorem 1
• Remark 1
• Theorem 2
• Remark 2
• Corollary 1
• Remark 3
• proof
• Lemma 1
• Lemma 2
• Lemma 3