Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the statistical efficiency of cross-conformal prediction

Matteo Gasparin, Aaditya Ramdas

TL;DR

This work addresses the statistical efficiency of cross-conformal prediction by introducing new variants that shrink prediction sets without sacrificing finite-sample marginal coverage. Building on exchangeable and randomized p-value combination results, the authors develop several methods, including e-mod-cross, u-mod-cross, and eu-mod-cross, that maintain a robust $1-2\alpha$ (or $1-2\alpha'$ under refinements) coverage while reducing set width. They provide theoretical guarantees and demonstrate through simulations and real-data applications that these variants can substantially decrease set size, with trade-offs in variability due to randomness and dependence. The results offer practical guidance for deploying conformal prediction in settings where computational efficiency and tighter uncertainty quantification are crucial, while preserving distribution-free validity.

Abstract

Vovk (2015) introduced cross-conformal prediction, a modification of split conformal designed to improve the width of prediction sets. The method, when trained with a miscoverage rate equal to $α$ and $n \gg K$, ensures a marginal coverage of at least $1 - 2α- 2(1-α)(K-1)/(n+K)$, where $n$ is the number of observations and $K$ denotes the number of folds. A simple modification of the method achieves coverage of at least $1-2α$. In this work, we propose new variants of both methods that yield smaller prediction sets without compromising the latter theoretical guarantees. The proposed methods are based on recent results deriving more statistically efficient combination of p-values that leverage exchangeability and randomization. Simulations confirm the theoretical findings and bring out some important tradeoffs.

Improving the statistical efficiency of cross-conformal prediction

TL;DR

This work addresses the statistical efficiency of cross-conformal prediction by introducing new variants that shrink prediction sets without sacrificing finite-sample marginal coverage. Building on exchangeable and randomized p-value combination results, the authors develop several methods, including e-mod-cross, u-mod-cross, and eu-mod-cross, that maintain a robust (or under refinements) coverage while reducing set width. They provide theoretical guarantees and demonstrate through simulations and real-data applications that these variants can substantially decrease set size, with trade-offs in variability due to randomness and dependence. The results offer practical guidance for deploying conformal prediction in settings where computational efficiency and tighter uncertainty quantification are crucial, while preserving distribution-free validity.

Abstract

Vovk (2015) introduced cross-conformal prediction, a modification of split conformal designed to improve the width of prediction sets. The method, when trained with a miscoverage rate equal to and , ensures a marginal coverage of at least , where is the number of observations and denotes the number of folds. A simple modification of the method achieves coverage of at least . In this work, we propose new variants of both methods that yield smaller prediction sets without compromising the latter theoretical guarantees. The proposed methods are based on recent results deriving more statistically efficient combination of p-values that leverage exchangeability and randomization. Simulations confirm the theoretical findings and bring out some important tradeoffs.

Paper Structure

This paper contains 26 sections, 7 theorems, 39 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Problem setup and related work
  3. Related work
  4. (Modified) Cross-conformal prediction
  5. Cross-conformal prediction
  6. Modified cross-conformal prediction
  7. New variants of cross-conformal prediction
  8. Exchangeable modified cross-conformal prediction
  9. Randomized modified cross-conformal prediction
  10. Exchangeable and randomized modified cross-conformal prediction
  11. Improving cross-conformal prediction
  12. Empirical results
  13. Simulation study
  14. Real data application
  15. Discussion
  16. ...and 11 more sections

Key Result

Proposition 4.1

Let $P_1(Y_{n+1}), \dots, P_K(Y_{n+1})$ be the (cross-conformal) p-values obtained using data $Z_i=(X_i,Y_i), \, i=[n+1],$ then $P_1(Y_{n+1}), \dots, P_K(Y_{n+1})$ are exchangeable, meaning that $\mathbf{P} \stackrel{d}{=} \mathbf{P}^\pi$, where $\stackrel{d}{=}$ represents equality in distribution,

Figures (6)

  • Figure 1: Simulation results, showing the size and coverage of the predictive sets for cross-conformal prediction and its variants. In the left plot, peaks are observed at $404,102,286,100$ and $307$ for mod-cross, e-mod-cross, u-mod-cross, eu-mod-cross and cross, respectively. The parameter $\alpha$ is set to 0.1. The smaller sets are often obtained using eu-mod-cross that has coverage between $1-2\alpha$ and $1-\alpha$. The randomized method (u-mod-cross) performs similarly to cross-conformal prediction.
  • Figure 2: Simulation results, showing the size and coverage of the predictive intervals obtained using $4$ methods. The parameter $\alpha$ is set to 0.1. Split conformal prediction is trained at levels $\alpha$ and $2\alpha$ and it is compared with mod-cross and eu-mod-cross. The modified cross-conformal prediction method always overcovers and tends to produce large prediction sets. Its exchangeable and randomized variant gives good results in terms of size. When $p\in[25,60]$, the average size of the eu-mod-cross method is smaller than that of the split conformal prediction method trained at level $2\alpha$.
  • Figure 3: Empirical size obtained using different regression algorithms and different conformal prediction methods. The methods mod-cross and cross give similar results. The variants that use randomization (u-mod-cross and eu-mod-cross) have a smaller size with respect to the other methods trained at level $\alpha$. The smaller sets are obtained using split conformal prediction trained at level $2\alpha$.
  • Figure 4: Comparison of the bounds in \ref{['eq:cov-cross-cp']} and \ref{['eq:cov_cv+']} for different values of $K$ with $\alpha = 0.1$. Dashed lines represent the levels $1-2\alpha-2/\sqrt{n}$ and $1-2\alpha$.
  • Figure 5: Simulation results, showing the size and the coverage of the predictive intervals for jackknife+, split conformal prediction and full conformal prediction. The eu-mod-cross method is added for comparison and the $\alpha$-level is set to $0.1$. The smaller sets are usually observed by eu-mod-cross conformal prediction. Split conformal prediction and jackknife+ have empirical coverage $\approx 1-\alpha$.
  • ...and 1 more figures

Theorems & Definitions (18)

  • Remark 3.1
  • Remark 3.2
  • Proposition 4.1
  • Lemma 4.2: dean1990kuchibhotla2020
  • Remark 4.3
  • Theorem 4.4
  • Remark 4.5
  • Theorem 4.6
  • Theorem 4.7
  • Remark 4.8: Randomization and "interval-hacking"
  • ...and 8 more