What Does It Take to Build a Performant Selective Classifier?

TL;DR

The paper formalizes a finite-sample selective-classification gap Δ(c) between real predictors and a perfect-ordering oracle, decomposing it into five components: Bayes noise, approximation error, ranking error, statistical noise, and miscellaneous slack. It shows monotone post-hoc calibration has limited impact on reducing the gap, emphasizing the need for score functions that actively reorder predictions using richer features or ensembles. Through synthetic two-moons experiments and real vision and language benchmarks, the authors isolate each error source, finding that Bayes noise and capacity limits dominate the gap while richer, non-monotone scoring improves rank-ordering under distribution shift. The work provides an actionable error budget and practical guidelines for designing selective classifiers that closely approximate ideal oracle behavior, with implications for reliability in high-stakes deployments.

Abstract

Selective classifiers improve model reliability by abstaining on inputs the model deems uncertain. However, few practical approaches achieve the gold-standard performance of a perfect-ordering oracle that accepts examples exactly in order of correctness. Our work formalizes this shortfall as the selective-classification gap and present the first finite-sample decomposition of this gap to five distinct sources of looseness: Bayes noise, approximation error, ranking error, statistical noise, and implementation- or shift-induced slack. Crucially, our analysis reveals that monotone post-hoc calibration -- often believed to strengthen selective classifiers -- has limited impact on closing this gap, since it rarely alters the model's underlying score ranking. Bridging the gap therefore requires scoring mechanisms that can effectively reorder predictions rather than merely rescale them. We validate our decomposition on synthetic two-moons data and on real-world vision and language benchmarks, isolating each error component through controlled experiments. Our results confirm that (i) Bayes noise and limited model capacity can account for substantial gaps, (ii) only richer, feature-aware calibrators meaningfully improve score ordering, and (iii) data shift introduces a separate slack that demands distributionally robust training. Together, our decomposition yields a quantitative error budget as well as actionable design guidelines that practitioners can use to build selective classifiers which approximate ideal oracle behavior more closely.

Figures (6)

• Figure 1: Visualization of the selective classification gap $\Delta(c)$. The dashed curve is the oracle frontier $\overline{\mathrm{acc}}(a_{\text{full}},c)$ under which coverage levels left of $c=a_{\text{full}}$ (blue) accept all correct predictions first, and rank incorrect predictions last (orange). This constitutes the ideal behavior of a selective predictor. On the other hand, the solid curve shows the realized selective accuracy $\mathrm{acc}_{c}(h,g)$. The mismatch between $\overline{\mathrm{acc}}(a_{\text{full}},c)$ and $\mathrm{acc}_{c}(h,g)$ at coverage $c$ is the gap $\Delta(c)$; the gray shaded area visualizes this gap over the full coverage spectrum.
• Figure 2: Experiments on approximation error. We find that approximation error is a major contributor to the gap. (a) We show the two moons dataset fitted with models of different degrees of expressiveness as well as the corresponding accuracy-coverage tradeoffs. (b) + (c) Accuracy-coverage tradeoffs for various model architectures on CIFAR-100 and StanfordCars, respectively.
• Figure 3: Experiments on Bayes error. We find that irreducible noise significantly contributes to the gap. (a) We show the two moons dataset with varying degrees of noise $\sigma \in \{0.1,0.33,0.66,1.5\}$ as well as the corresponding accuracy-coverage tradeoffs. (b) + (c) Accuracy-coverage tradeoffs for the 10% (blue), 25% (orange), and 50% (green) most noisy images in CIFAR-10N/100N, respectively.
• Figure 4: Experiments on distribution shifts. We find that shifts can also significantly contribute to the gap. (a) Two moons under shear, rotation, and translation with corresponding accuracy–coverage curves. (b) CIFAR-10C across three distinct corruption severities. (c) Camelyon17 OOD shift.
• Figure 5: Comparison between gap and calibration on CIFAR-100.Top: selective accuracy curves across four training methods and three architectures. Bottom: corresponding reliability diagrams (ECE shown in parentheses). Temperature scaling (TEMP) consistently improves calibration but does not reduce the gap. By contrast, SAT and DE reduce the gap more effectively—especially for larger models—by improving the ranking.

Theorems & Definitions (15)

• Definition 1: Selective Classifier chow1957optimumel2010foundations
• Definition 2: Empirical Accuracy–Coverage Tradeoff
• Definition 3: Perfect Ordering Upper Bound geifman2018biasrabanser2023training
• Definition 4: Selective Classification Gap
• Remark : Distance to a Perfect Ranker
• Theorem 1: Selective Gap Bound
• Proof 1
• Definition 5: Perfect Calibration
• Theorem 2: Selective classification Gap; detailed
• Proof 2