Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evidential Uncertainty Sets in Deep Classifiers Using Conformal Prediction

Hamed Karimi, Reza Samavi

TL;DR

Evidential Conformal Prediction (ECP) fuses Conformal Prediction with Evidential Deep Learning to produce prediction sets that are both efficient and adaptive while guaranteeing marginal coverage. The approach derives a novel non-conformity score from logit-based evidential quantities, including belief masses, Dirichlet parameters, and a model uncertainty u, and uses uncertainty surprisal and expected utility to form per-label costs. ECP offers theoretical coverage guarantees and reliability metrics for marginal coverage, and experiments on ImageNet datasets show smaller prediction sets and improved adaptivity compared to APS and RAPS, without extra hyperparameter tuning. The work presents a practical, principled framework for uncertainty-aware classification with strong performance guarantees in safety-critical settings.

Abstract

In this paper, we propose Evidential Conformal Prediction (ECP) method for image classifiers to generate the conformal prediction sets. Our method is designed based on a non-conformity score function that has its roots in Evidential Deep Learning (EDL) as a method of quantifying model (epistemic) uncertainty in DNN classifiers. We use evidence that are derived from the logit values of target labels to compute the components of our non-conformity score function: the heuristic notion of uncertainty in CP, uncertainty surprisal, and expected utility. Our extensive experimental evaluation demonstrates that ECP outperforms three state-of-the-art methods for generating CP sets, in terms of their set sizes and adaptivity while maintaining the coverage of true labels.

Evidential Uncertainty Sets in Deep Classifiers Using Conformal Prediction

TL;DR

Evidential Conformal Prediction (ECP) fuses Conformal Prediction with Evidential Deep Learning to produce prediction sets that are both efficient and adaptive while guaranteeing marginal coverage. The approach derives a novel non-conformity score from logit-based evidential quantities, including belief masses, Dirichlet parameters, and a model uncertainty u, and uses uncertainty surprisal and expected utility to form per-label costs. ECP offers theoretical coverage guarantees and reliability metrics for marginal coverage, and experiments on ImageNet datasets show smaller prediction sets and improved adaptivity compared to APS and RAPS, without extra hyperparameter tuning. The work presents a practical, principled framework for uncertainty-aware classification with strong performance guarantees in safety-critical settings.

Abstract

In this paper, we propose Evidential Conformal Prediction (ECP) method for image classifiers to generate the conformal prediction sets. Our method is designed based on a non-conformity score function that has its roots in Evidential Deep Learning (EDL) as a method of quantifying model (epistemic) uncertainty in DNN classifiers. We use evidence that are derived from the logit values of target labels to compute the components of our non-conformity score function: the heuristic notion of uncertainty in CP, uncertainty surprisal, and expected utility. Our extensive experimental evaluation demonstrates that ECP outperforms three state-of-the-art methods for generating CP sets, in terms of their set sizes and adaptivity while maintaining the coverage of true labels.
Paper Structure (17 sections, 2 theorems, 21 equations, 3 figures, 8 tables)

This paper contains 17 sections, 2 theorems, 21 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evidential Conformal Prediction
  4. Evidential Deep Learning
  5. Evidential Classification Cost
  6. Evidential Prediction Set
  7. Insights on Evidential Conformal Prediction
  8. Reliability in Marginal Coverage
  9. Experimental Evaluations
  10. Experiment 1: Marginal Coverage and Prediction Set Size
  11. Experiment 2: Coverage in Size-stratified Prediction Sets
  12. Experiment 3: Adaptivity in Prediction Sets
  13. Experiment 4: Adaptivity in Conditional Coverage
  14. Conclusion
  15. Proof of conf_cov_theorem
  16. ...and 2 more sections

Key Result

theorem 1

Consider $\mathcal{X}_{cal}$ and $(x_{val},y_{val}) \in \mathcal{X}_{val}$ are i.i.d. and correspond to unseen test data as holdout data and a validation data point, respectively. Let $\delta$ be the user-chosen coverage error level, $\widehat{q}$ is the $1-\delta$ quantile of holdout non-conformity

Figures (3)

  • Figure 1: An overview of Evidential Deep Learning (EDL): For each data point, the evidence values are calculated based on logits to produce belief masses and Dirichlet parameters, and finally, achieving predictive probabilities.
  • Figure 2: Comparing ECP with SoTA methods using ResNet-152 with different choices of $\delta=\{0.1,0.05,0.01\}$ in terms of the average marginal coverage and prediction set size. For RAPS, we used $k_{reg}=5$ and $\lambda=0.1$.
  • Figure 3: Adaptivity in ECP compared to SoTA methods based on difficulty level for both test datasets using ResNet-152 with $\delta=0.1$. For RAPS, we used $k_{reg}=5$ and $\lambda=0.1$.

Theorems & Definitions (9)

  • definition 1: Evidence in DNN
  • definition 2: Base Rate (Prior Probability) in DNN
  • definition 3: Heuristic Model Uncertainty
  • theorem 1: Conformal Coverage Guarantee
  • definition 4: Evidential Classification Cost
  • definition 5: Uncertainty Surprisal
  • definition 6: Expected Utility
  • theorem 2: Coverage Confidence Guarantee
  • proof