Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty Sets for Image Classifiers using Conformal Prediction

Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, Michael I. Jordan

TL;DR

The paper addresses the challenge of quantifying uncertainty for image classifiers by wrapping any pre-trained model with conformal prediction to produce predictive sets that cover the true label with a user-specified probability. It introduces Regularized Adaptive Prediction Sets (RAPS), a simple, fast modification of Adaptive Prediction Sets (APS) that regularizes tail probabilities to yield smaller, more stable sets while maintaining finite-sample coverage guarantees. The approach is backed by theory showing conformal calibration guarantees and an optimality result relative to top-k schemes, and is validated on Imagenet and Imagenet-V2, where RAPS consistently reduces set size compared with naive and APS baselines. The work further develops adaptiveness metrics and automatic parameter tuning, arguing that RAPS provides a practical, scalable uncertainty quantification tool for high-dimensional image classification tasks with potential use in critical decision-making contexts.

Abstract

Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving coverage with sets that are often factors of 5 to 10 smaller than a stand-alone Platt scaling baseline.

Uncertainty Sets for Image Classifiers using Conformal Prediction

TL;DR

The paper addresses the challenge of quantifying uncertainty for image classifiers by wrapping any pre-trained model with conformal prediction to produce predictive sets that cover the true label with a user-specified probability. It introduces Regularized Adaptive Prediction Sets (RAPS), a simple, fast modification of Adaptive Prediction Sets (APS) that regularizes tail probabilities to yield smaller, more stable sets while maintaining finite-sample coverage guarantees. The approach is backed by theory showing conformal calibration guarantees and an optimality result relative to top-k schemes, and is validated on Imagenet and Imagenet-V2, where RAPS consistently reduces set size compared with naive and APS baselines. The work further develops adaptiveness metrics and automatic parameter tuning, arguing that RAPS provides a practical, scalable uncertainty quantification tool for high-dimensional image classification tasks with potential use in critical decision-making contexts.

Abstract

Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Our method modifies an existing conformal prediction algorithm to give more stable predictive sets by regularizing the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving coverage with sets that are often factors of 5 to 10 smaller than a stand-alone Platt scaling baseline.

Paper Structure

This paper contains 24 sections, 4 theorems, 17 equations, 4 figures, 11 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Conformal calibration
  5. Our method
  6. Why regularize?
  7. Optimality considerations
  8. Experiments
  9. Experiment 1: coverage vs set size on Imagenet
  10. Experiment 2: coverage vs set size on Imagenet-V2
  11. Experiment 3: Set sizes of naive, APS, and RAPS on Imagenet
  12. Experiment 4: Adaptiveness of RAPS on Imagenet
  13. Experiment 5: choice of tuning parameters
  14. Adaptiveness and conditional coverage
  15. A metric of conditional coverage
  16. ...and 9 more sections

Key Result

Theorem 1

Suppose $(X_i, Y_i, U_i)_{i = 1,\dots,n}$ and $(X_{n+1}, Y_{n+1}, U_{n+1})$ are i.i.d. and let $\mathcal{C}(x, u, \tau)$ be a set-valued function satisfying the nesting property in Eq. (eq:nested_sets). Suppose further that the sets $\mathcal{C}(x,u,\tau)$ grow to include all labels for large enough

Figures (4)

  • Figure 1: Prediction set examples on Imagenet. We show three examples of the class fox squirrel and the $95\%$ prediction sets generated by RAPS to illustrate how the size of the set changes as a function of the difficulty of a test-time image.
  • Figure 2: Coverage and average set size on Imagenet for prediction sets from three methods. All methods use a ResNet-152 as the base classifier, and results are reported for 100 random splits of Imagenet-Val, each of size 20K. See Section \ref{['subsec:imagenet-val']} for full details.
  • Figure 3: Visualizations of conformal calibration and RAPS sets. In the left panel, the y-axis shows the empirical coverage on the conformal calibration set, and $1-\alpha' = \lceil(n+1)(1-\alpha)\rceil / n$. In the right panel, the printed numbers indicate the cumulative probability plus penalty mass. For the indicated value $\hat{\tau}_\textnormal{ccal}$, the RAPS prediction set is {c, d, f, b}.
  • Figure 4: Set sizes produced with ResNet-152. See Section \ref{['subsec:size-histograms']} for details.

Theorems & Definitions (8)

  • Theorem 1: Conformal calibration coverage guarantee
  • Proposition 1: RAPS coverage guarantee
  • Proposition 2: RAPS dominates top-k sets
  • Proposition 3
  • proof : Theorem \ref{['thm:conformal_calibration']}
  • proof : Proposition \ref{['prop:raps_coverage']}
  • proof : Proposition \ref{['prop:topk']}
  • proof : Proposition \ref{['prop:conditional_covg_size']}