E-Values Expand the Scope of Conformal Prediction

TL;DR

This work extends conformal prediction by replacing p-values with e-values to form conformal e-prediction, enabling sequential, data-adaptive, and non-exchangeable uncertainty quantification. It develops three applications—batch anytime-valid conformal prediction, fixed-size conformal sets with data-dependent coverage, and conformal prediction under ambiguous ground truth—grounded in e-variables and Ville-type inequalities. The authors provide theoretical frameworks and practical demonstrations on FEMNIST and CIFAR datasets, showing improved flexibility and robustness, including sequential guarantees and post-hoc coverage control. The findings suggest that conformal e-prediction broadens the utility of uncertainty quantification in complex, dynamic ML settings, with opportunities for further score-function design and tighter sequential guarantees.

Abstract

Conformal prediction is a powerful framework for distribution-free uncertainty quantification. The standard approach to conformal prediction relies on comparing the ranks of prediction scores: under exchangeability, the rank of a future test point cannot be too extreme relative to a calibration set. This rank-based method can be reformulated in terms of p-values. In this paper, we explore an alternative approach based on e-values, known as conformal e-prediction. E-values offer key advantages that cannot be achieved with p-values, enabling new theoretical and practical capabilities. In particular, we present three applications that leverage the unique strengths of e-values: batch anytime-valid conformal prediction, fixed-size conformal sets with data-dependent coverage, and conformal prediction under ambiguous ground truth. Overall, these examples demonstrate that e-value-based constructions provide a flexible expansion of the toolbox of conformal prediction.

Figures (20)

• Figure 1: Some images from FEMNIST with their associated class label.
• Figure 2: Histogram of conformal set sizes obtained across all $T=50$ data batches and 100 iterations. The distribution illustrates the variability in set sizes, highlighting the proportion of informative and trivial sets.
• Figure 3: Histogram of $1-\tilde{\alpha}$ for $C = 3$ and $C = 5$, computed across 100 iterations. The black dashed line represents the expected coverage level, $1-\mathbb{E}[\tilde{\alpha}]$, while the red dashed line corresponds to the empirical coverage probability, $\mathbb{P}(Y_{n+1} \in \hat{C}_n^{\tilde{\alpha}}(X_{n+1}))$, both estimated over the 100 iterations.
• Figure 4: Example of conformal sets obtained with varying $\tilde{\alpha}$.
• Figure 5: Some images from CIFAR-10H with ambiguous ground truth, along with their label distributions.

Theorems & Definitions (24)

• Lemma 1.1
• proof
• Definition 1.2: e-variable
• Remark 1.3: Terminology
• Definition 2.1: Filtration
• Definition 2.2: Martingale
• Theorem 2.3: Ville's Inequality
• Remark 2.4: Ville’s Inequality with Stopping Times
• Theorem 2.5
• proof