Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Validation Conformal Risk Control

Kfir M. Cohen, Sangwoo Park, Osvaldo Simeone, Shlomo Shamai

TL;DR

The paper addresses uncertainty quantification for predictive sets under risk constraints by extending CRC to a cross-validation framework. It introduces CV-CRC, which uses K-fold cross-validation and a minimum nonconformity score across folds to form predictive sets, with a threshold chosen to bound the average risk by $\alpha$; theoretical guarantees require bounded, monotone losses and a permutation-invariant nonconformity score. The method generalizes CV-CP to arbitrary bounded risks and shows reduced average set sizes compared to VB-CRC when data are scarce, demonstrated on vector regression and temporal point process tasks. This cross-validated approach enables more data-efficient, calibration-guaranteed uncertainty quantification with potential extensions via jackknife+ and meta-learning.

Abstract

Conformal risk control (CRC) is a recently proposed technique that applies post-hoc to a conventional point predictor to provide calibration guarantees. Generalizing conformal prediction (CP), with CRC, calibration is ensured for a set predictor that is extracted from the point predictor to control a risk function such as the probability of miscoverage or the false negative rate. The original CRC requires the available data set to be split between training and validation data sets. This can be problematic when data availability is limited, resulting in inefficient set predictors. In this paper, a novel CRC method is introduced that is based on cross-validation, rather than on validation as the original CRC. The proposed cross-validation CRC (CV-CRC) extends a version of the jackknife-minmax from CP to CRC, allowing for the control of a broader range of risk functions. CV-CRC is proved to offer theoretical guarantees on the average risk of the set predictor. Furthermore, numerical experiments show that CV-CRC can reduce the average set size with respect to CRC when the available data are limited.

Cross-Validation Conformal Risk Control

TL;DR

The paper addresses uncertainty quantification for predictive sets under risk constraints by extending CRC to a cross-validation framework. It introduces CV-CRC, which uses K-fold cross-validation and a minimum nonconformity score across folds to form predictive sets, with a threshold chosen to bound the average risk by ; theoretical guarantees require bounded, monotone losses and a permutation-invariant nonconformity score. The method generalizes CV-CP to arbitrary bounded risks and shows reduced average set sizes compared to VB-CRC when data are scarce, demonstrated on vector regression and temporal point process tasks. This cross-validated approach enables more data-efficient, calibration-guaranteed uncertainty quantification with potential extensions via jackknife+ and meta-learning.

Abstract

Conformal risk control (CRC) is a recently proposed technique that applies post-hoc to a conventional point predictor to provide calibration guarantees. Generalizing conformal prediction (CP), with CRC, calibration is ensured for a set predictor that is extracted from the point predictor to control a risk function such as the probability of miscoverage or the false negative rate. The original CRC requires the available data set to be split between training and validation data sets. This can be problematic when data availability is limited, resulting in inefficient set predictors. In this paper, a novel CRC method is introduced that is based on cross-validation, rather than on validation as the original CRC. The proposed cross-validation CRC (CV-CRC) extends a version of the jackknife-minmax from CP to CRC, allowing for the control of a broader range of risk functions. CV-CRC is proved to offer theoretical guarantees on the average risk of the set predictor. Furthermore, numerical experiments show that CV-CRC can reduce the average set size with respect to CRC when the available data are limited.
Paper Structure (14 sections, 5 theorems, 46 equations, 5 figures)

This paper contains 14 sections, 5 theorems, 46 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Context and Motivation
  3. State of the Art
  4. Main Contributions
  5. Background
  6. Cross-Validation Conformal Risk Control
  7. Examples
  8. Vector Regression
  9. Temporal Point Process Prediction
  10. Conclusion
  11. Proof that VB-CRC achieves target risk
  12. Proof of Lemma \ref{['lemma: conditional expectation is empirical mean']}
  13. Proof of Theorem \ref{['thm: K-CV-CRC']}
  14. Proof of Lemma \ref{['lemma: lambdaL2O <= lambda']}

Key Result

Theorem 1

Fix any bounded and monotonic loss function $\ell(\cdot,\cdot)$ satisfying conditions eq: ell <= B and eq: nesting loss, and any NC score ${\mathrm{NC}}((x,y)|{\mathcal{D}^\text{tr}})$ that is permutation-invariant with respect to the ordering of the examples in the training set ${\mathcal{D}^\text{

Figures (5)

  • Figure 1: Illustration of (top) the existing validation-based conformal risk control (VB-CRC) angelopoulos2022conformal; and (bottom) the proposed method cross-validation-based conformal risk control (CV-CRC), which aims at reducing the predictive sets sizes by reusing the available data ${\mathcal{D}}$ more efficiently.
  • Figure 2: Empirical risk of VB-CRC and CV-CRC for the vector regression problem.
  • Figure 3: Empirical inefficiency of VB-CRC and CV-CRC for the vector regression problem.
  • Figure 4: Temporal point process prediction: After observing the past $d$ times $t_1,\dots,t_n$, a point process set predictor outputs predictive intervals $\Gamma_j(x|{\mathcal{D}})$ for each of the next $m$ points with $j=1,\dots,m$.
  • Figure 5: Empirical risk (top) and inefficiency (bottom) of VB-CRC and $N$-CV-CRC for the temporal point process prediction problem.

Theorems & Definitions (5)

  • Theorem 1
  • Lemma 2
  • Lemma 3
  • Corollary 4
  • Lemma 5