Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fuzzy Prediction Sets: Conformal Prediction with E-values

Nick W. Koning, Sam van Meer

TL;DR

This work extends conformal prediction by introducing fuzzy prediction sets that assign a degree of exclusion to each potential next observation, and ties these to e-values to provide an interpretable, data-dependent evidence scale. By framing prediction sets as tests and leveraging an expected-utility approach, the authors derive optimal fuzzy prediction sets and show how classical conformal prediction is recovered as a special case. The paper unifies predictive validity, optimality, and decision-making under a single e-value framework, and demonstrates extensions to covariates and practical applications (e.g., image recognition with FEMNIST). The resulting methodology supports merging, post-hoc level selection, and weighted-loss decisions, enabling robust, certifiable predictions across models and data contexts.

Abstract

We make three contributions to conformal prediction. First, we propose fuzzy conformal prediction sets that offer a degree of exclusion, generalizing beyond the binary inclusion/exclusion offered by classical prediction sets. We connect fuzzy prediction sets to e-values to show this degree of exclusion is equivalent to an exclusion at different confidence levels, capturing precisely what e-values bring to conformal prediction. We show that a fuzzy prediction set is a predictive distribution with an arguably more appropriate error guarantee. Second, we derive optimal conformal prediction sets by interpreting the minimization of the expected measure of a prediction set as an optimal testing problem against a particular alternative. We use this to characterize exactly in what sense traditional conformal prediction is optimal, and show how this may generally be used to construct optimal (fuzzy) prediction sets. Third, we generalize the inheritance of guarantees by subsequent minimax decisions from prediction sets to fuzzy prediction sets. All results generalize beyond the conformal setting to prediction sets for arbitrary models. In particular, we find that constructing a (fuzzy) prediction set for a model is equivalent to constructing a test (e-value) for that model as a hypothesis.

Fuzzy Prediction Sets: Conformal Prediction with E-values

TL;DR

This work extends conformal prediction by introducing fuzzy prediction sets that assign a degree of exclusion to each potential next observation, and ties these to e-values to provide an interpretable, data-dependent evidence scale. By framing prediction sets as tests and leveraging an expected-utility approach, the authors derive optimal fuzzy prediction sets and show how classical conformal prediction is recovered as a special case. The paper unifies predictive validity, optimality, and decision-making under a single e-value framework, and demonstrates extensions to covariates and practical applications (e.g., image recognition with FEMNIST). The resulting methodology supports merging, post-hoc level selection, and weighted-loss decisions, enabling robust, certifiable predictions across models and data contexts.

Abstract

We make three contributions to conformal prediction. First, we propose fuzzy conformal prediction sets that offer a degree of exclusion, generalizing beyond the binary inclusion/exclusion offered by classical prediction sets. We connect fuzzy prediction sets to e-values to show this degree of exclusion is equivalent to an exclusion at different confidence levels, capturing precisely what e-values bring to conformal prediction. We show that a fuzzy prediction set is a predictive distribution with an arguably more appropriate error guarantee. Second, we derive optimal conformal prediction sets by interpreting the minimization of the expected measure of a prediction set as an optimal testing problem against a particular alternative. We use this to characterize exactly in what sense traditional conformal prediction is optimal, and show how this may generally be used to construct optimal (fuzzy) prediction sets. Third, we generalize the inheritance of guarantees by subsequent minimax decisions from prediction sets to fuzzy prediction sets. All results generalize beyond the conformal setting to prediction sets for arbitrary models. In particular, we find that constructing a (fuzzy) prediction set for a model is equivalent to constructing a test (e-value) for that model as a hypothesis.

Paper Structure

This paper contains 42 sections, 15 theorems, 95 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Traditional conformal prediction
  3. Fuzzy conformal prediction
  4. E-values and fuzzy prediction
  5. Optimal (fuzzy) prediction sets
  6. Certifying subsequent decisions
  7. Covariates and application
  8. Related literature
  9. Prediction sets
  10. Slicing a prediction set for $Z^{n+1}$
  11. Prediction as a testing problem
  12. Optimal prediction sets and power
  13. Handling unknown $P_*^{Z^n}$
  14. Examples: Gaussian location
  15. Fuzzy prediction sets
  16. ...and 27 more sections

Key Result

Theorem 1

A prediction set $C_\alpha^{Z^n}$ is equivalent to the test $C_\alpha$, The prediction set $C_\alpha^{Z^n}$ is valid at level $\alpha$ for the model $\mathcal{P}$ if and only if this test $C_\alpha$ is valid at level $\alpha$ for the hypothesis $\mathcal{P}$.

Figures (6)

  • Figure 1: A traditional and fuzzy conformal prediction set. The solid line represents a traditional level $\alpha = 0.05$ prediction set: the points $z_{n+1}$ at which this line is zero form the prediction set, and the points at which it equals $1$ its complement. The dashed line represents a fuzzy $[0, 1]$-valued prediction set, offering a degree of exclusion at each point.
  • Figure 2: One traditional and two fuzzy prediction sets on the evidence scale. The solid line represents a traditional level $\alpha = 0.05$ prediction set, the dashed line represents a fuzzy $[0, 1/\alpha]$-valued prediction set, and the dotted line a fuzzy $[0, \infty]$-valued prediction set. An evidence value of $\mathcal{E}^{Z^n}(z) = a$ corresponds to an exclusion at significance level $1/a$.
  • Figure 3: Abstract illustration of the sample space $\mathcal{Z}^{n+1}$, where the horizontal dimension represents $\mathcal{Z}^n$ and the vertical dimension $\mathcal{Z}_{n+1}$. The prediction set $C_\alpha$ for $Z^{n+1}$ is shaded white, and its complement $\mathcal{Z}^{n+1} \setminus C_\alpha$ grey. The prediction set $C_\alpha^{Z^n}$ for $Z_{n+1}$ appears as a $Z^n$-dependent random slice of $C_\alpha$.
  • Figure 4: Optimal fuzzy prediction sets for Neyman--Pearson-utility (solid), log-utility (dotted) and bounded log-utility (dashed) under the simple Gaussian setting (left) and composite Gaussian setting (right) from Example \ref{['exm:simple_gaussian_NP']}, \ref{['exm:composite_gaussian_NP']} and \ref{['exm:fuzzy_gaussian']}, for $n = 3$, $\mu = 0$, $\sigma = 1$, $\tau = 3.5$, $\bar{Z}_n = 1.44$. Here, the Neyman--Pearson utility and bounded log-utility are both bounded at $1/0.05 = 20$.
  • Figure 5: Expected-power-utility-optimal $[0, 1/\alpha]$-valued fuzzy prediction sets over possible labels, for $h = 0$ (log-utility), $h = 1/2$ (power-utility) and $h = 1$ (Neyman--Pearson), for values $\alpha = 0$ (top-left) and $\alpha = 0.001$ (others). The true character label is marked in red (d).
  • ...and 1 more figures

Theorems & Definitions (31)

  • Remark 1: Slicing on observables
  • Theorem 1
  • Remark 2: Connection to classical testing
  • Theorem 2
  • proof
  • Example 1: i.i.d. simple Gaussian
  • Example 2: i.i.d. Composite Gaussian
  • Example 3: Autoregressive Gaussian
  • Example 4: Prediction set based on two-sided z-test
  • Remark 3: Randomization
  • ...and 21 more