Quantifying Epistemic Predictive Uncertainty in Conformal Prediction

TL;DR

This paper addresses how to quantify epistemic predictive uncertainty (EPU) within conformal prediction (CP). By showing that split CP inherently induces a predictive credal set and establishing a link to imprecise probabilities, the authors adapt the Maximum Mean Imprecision (MMI) framework to CP, yielding the MMI-CP measure. They derive closed-form, computationally efficient expressions for EPU in both classification and regression contexts, and demonstrate through active learning and selective classification experiments that MMI-CP provides more informative uncertainty assessments than CPR size alone. The work offers a principled basis for decision-making under epistemic uncertainty in CP and clarifies the role of consonance assumptions in linking CP to imprecise probability formalisms.

Abstract

We study the problem of quantifying epistemic predictive uncertainty (EPU) -- that is, uncertainty faced at prediction time due to the existence of multiple plausible predictive models -- within the framework of conformal prediction (CP). To expose the implicit model multiplicity underlying CP, we build on recent results showing that, under a mild assumption, any full CP procedure induces a set of closed and convex predictive distributions, commonly referred to as a credal set. Importantly, the conformal prediction region (CPR) coincides exactly with the set of labels to which all distributions in the induced credal set assign probability at least $1-α$. As our first contribution, we prove that this characterisation also holds in split CP. Building on this connection, we then propose a computationally efficient and analytically tractable uncertainty measure, based on \emph{Maximum Mean Imprecision}, to quantify the EPU by measuring the degree of conflicting information within the induced credal set. Experiments on active learning and selective classification demonstrate that the quantified EPU provides substantially more informative and fine-grained uncertainty assessments than reliance on CPR size alone. More broadly, this work highlights the potential of CP serving as a principled basis for decision-making under epistemic uncertainty.

Figures (7)

• Figure 1: Although the two instances share the same prediction set, the left instance is evidently more certain, as reflected by uniformly smaller conformal p-values for labels $(1,2,3)$. How can this intuition be formalised?
• Figure 2: Instead of distribution of plausible models as in Bayesian prediction, conformal prediction implicitly yields a set of plausible models as shown by cella. How can we quantify the epistemic predictive uncertainty in this case?
• Figure 3: Conformal p-value portfolio for classification and regression, where each colour represents a different test instance. While conformal classification exhibits instance-specific p-value distributions, conformal regression yields distributions with identical shapes across instances.
• Figure 4: Results averaged over 10 seeds and 1 standard error reported. In general, MMI-based approaches outperform the set-size-based approaches, suggesting that the former provide more fine-grained epistemic uncertainty information for downstream decision-making.
• Figure 5: Examining how the choice of nonconformity score affects performance in selective classification on the CIFAR-100 dataset.

Theorems & Definitions (30)

• Definition 2.1: Capacities and Upper probabilities cerreia2016ergodic
• Definition 2.2: Core
• Definition 2.3: MMI
• Definition 3.1: Consonance
• Proposition 3.1
• Definition 3.2: Imprecise Highest Density Region coolen
• Proposition 3.2
• Proposition 3.2
• Proposition 3.2
• Proposition 3.2