Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Calibrating Conformal Prediction

Lars van der Laan, Ahmed M. Alaa

TL;DR

Self-Calibrating Conformal Prediction is introduced, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.

Abstract

In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making. Recognizing the complementary roles of point and interval predictions, we introduce Self-Calibrating Conformal Prediction, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions. To achieve this, we extend the original Venn-Abers procedure from binary classification to regression. Our theoretical framework supports analyzing conformal prediction methods that involve calibrating model predictions and subsequently constructing conditionally valid prediction intervals on the same data, where the conditioning set or conformity scores may depend on the calibrated predictions. Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.

Self-Calibrating Conformal Prediction

TL;DR

Self-Calibrating Conformal Prediction is introduced, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.

Abstract

In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making. Recognizing the complementary roles of point and interval predictions, we introduce Self-Calibrating Conformal Prediction, a method that combines Venn-Abers calibration and conformal prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions. To achieve this, we extend the original Venn-Abers procedure from binary classification to regression. Our theoretical framework supports analyzing conformal prediction methods that involve calibrating model predictions and subsequently constructing conditionally valid prediction intervals on the same data, where the conditioning set or conformity scores may depend on the calibrated predictions. Real-data experiments show that our method improves interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.
Paper Structure (24 sections, 3 theorems, 59 equations, 11 figures, 2 algorithms)

This paper contains 24 sections, 3 theorems, 59 equations, 11 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem setup
  3. Notation
  4. Conditional predictive inference and a curse of dimensionality
  5. A dual calibration objective
  6. Self-Calibrating Conformal Prediction
  7. Preliminaries on point calibration
  8. Venn-Abers calibration
  9. Conformalizing Venn-Abers Calibration
  10. Computational considerations
  11. Theoretical guarantees
  12. Related work
  13. Real-Data Experiments: predicting utilization of medical services
  14. Experimental setup
  15. Results and discussion
  16. ...and 9 more sections

Key Result

Theorem 4.1

Under Conditions cond::exchange and cond::variance, the Venn-Abers multi-prediction $f_{n, X_{n+1}}(X_{n+1})$ almost surely satisfies the condition $f_n^{(X_{n+1}, Y_{n+1})}(X_{n+1}) = \mathbb{E}[Y_{n+1} \mid f_n^{(X_{n+1}, Y_{n+1})}(X_{n+1})]$.

Figures (11)

  • Figure 1: Example SC-CP output with small $\mathcal{C}_n$ ($n = 200$).
  • Figure 2: MEPS-21 dataset: Calibration plot for SC-CP, prediction bands for SC-CP and baselines, and empirical coverage, width, and calibration error within sensitive subgroup.
  • Figure 3: STAR dataset: Calibration plot for SC-CP, prediction bands for SC-CP and baselines, and empirical coverage, width, and calibration error within sensitive subgroup.
  • Figure 4: Bike dataset: Calibration plot for SC-CP, prediction bands for SC-CP and baselines, and empirical coverage, width, and calibration error within sensitive subgroup.
  • Figure 5: Community dataset: Calibration plot for SC-CP, prediction bands for SC-CP and baselines, and empirical coverage, width, and calibration error within sensitive subgroup.
  • ...and 6 more figures

Theorems & Definitions (6)

  • Theorem 4.1: Perfect calibration of Venn-Abers multi-prediction
  • Theorem 4.2: Self-calibration of prediction interval
  • Theorem 4.3
  • proof : Proof of Theorem \ref{['theorem::point']}
  • proof : Proof of Theorem \ref{['theorem::coverage']}
  • proof : Proof of Theorem \ref{['theorem::interaction']}