Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Conformal Prediction Using E-Test Statistics

A. A. Balinsky, A. D. Balinsky

TL;DR

This work tackles uncertainty quantification in machine learning predictions via Conformal Prediction (CP) under exchangeability, and proposes enhancing CP with e-test statistics using a BB-predictor. The core theoretical contribution shows that for exchangeable non-negative scores $L_i$, the ratio $F = \frac{L_{n+1}}{(\sum_{j=1}^{n+1} L_j)/(n+1)}$ satisfies $\mathbf{E}(F)=1$ and $P(F \ge 1/\alpha) \le \alpha$, enabling a BB-predictor based bound on $L_{n+1}$ and thus a distribution-free conformal guarantee. The paper complements theory with MNIST-based experiments comparing e-test statistics to p-value statistics in inductive CP, highlighting distinct label-allocations and validating the practical utility of the BB-predictor approach. Overall, it offers a principled, distribution-free mechanism for reliable uncertainty quantification in prediction intervals, with potential impact on vision tasks and beyond, by broadening the CP toolkit beyond p-values to e-test based bounds.

Abstract

Conformal Prediction (CP) serves as a robust framework that quantifies uncertainty in predictions made by Machine Learning (ML) models. Unlike traditional point predictors, CP generates statistically valid prediction regions, also known as prediction intervals, based on the assumption of data exchangeability. Typically, the construction of conformal predictions hinges on p-values. This paper, however, ventures down an alternative path, harnessing the power of e-test statistics to augment the efficacy of conformal predictions by introducing a BB-predictor (bounded from the below predictor).

Enhancing Conformal Prediction Using E-Test Statistics

TL;DR

This work tackles uncertainty quantification in machine learning predictions via Conformal Prediction (CP) under exchangeability, and proposes enhancing CP with e-test statistics using a BB-predictor. The core theoretical contribution shows that for exchangeable non-negative scores , the ratio satisfies and , enabling a BB-predictor based bound on and thus a distribution-free conformal guarantee. The paper complements theory with MNIST-based experiments comparing e-test statistics to p-value statistics in inductive CP, highlighting distinct label-allocations and validating the practical utility of the BB-predictor approach. Overall, it offers a principled, distribution-free mechanism for reliable uncertainty quantification in prediction intervals, with potential impact on vision tasks and beyond, by broadening the CP toolkit beyond p-values to e-test based bounds.

Abstract

Conformal Prediction (CP) serves as a robust framework that quantifies uncertainty in predictions made by Machine Learning (ML) models. Unlike traditional point predictors, CP generates statistically valid prediction regions, also known as prediction intervals, based on the assumption of data exchangeability. Typically, the construction of conformal predictions hinges on p-values. This paper, however, ventures down an alternative path, harnessing the power of e-test statistics to augment the efficacy of conformal predictions by introducing a BB-predictor (bounded from the below predictor).
Paper Structure (6 sections, 3 theorems, 16 equations, 6 figures)

This paper contains 6 sections, 3 theorems, 16 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Main Result
  3. Experiments and Results
  4. Inductive Conformal Prediction with e-test statistics
  5. Inductive Conformal Prediction with p-value statistics
  6. Conclusion

Key Result

Lemma 1

Suppose $L_1, \ldots, L_{n+1}$ are exchangeable random variables. Set i.e., the number of $L$ that are at least as large as the last one. Then for all $\epsilon\in [0,1]$

Figures (6)

  • Figure 1: The first nine images from the MNIST database of handwritten digits.
  • Figure 2: Our model.
  • Figure 3: Some images that the model predicts the wrong labels.
  • Figure 4: Values of the loss function on the CalibrationSet.
  • Figure 5: Some images that have multiple labels prediction with their labels.
  • ...and 1 more figures

Theorems & Definitions (5)

  • Lemma 1
  • Theorem 1
  • proof
  • Corollary 1.1
  • Definition