Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leave-One-Out Stable Conformal Prediction

Kiljae Lee, Yuan Zhang

TL;DR

This work addresses the computational bottleneck of full conformal prediction when multiple predictions are required by introducing Leave-One-Out Stable Conformal Prediction (LOO-StabCP), which uses leave-one-out stability to correct a single model fit on the training data. The method preserves finite-sample coverage $\mathbb{P}(Y_{n+j} \in \mathcal{C}^{\mathrm{LOO}}_{j,\alpha}(X_{n+j})) \ge 1-\alpha$ while dramatically reducing the need to refit models for each test point, outperforming RO-StabCP in large-scale prediction tasks. The authors derive LOO stability bounds for Regularized Loss Minimization (RLM) and Stochastic Gradient Descent (SGD), extend the framework to kernel methods, neural networks, and bagging, and validate the approach through simulations and real data, including a conformalized screening application (LOO-cfBH) that improves power under FDR control. Overall, LOO-StabCP enables scalable, distribution-free uncertainty quantification with strong practical impact for large-scale prediction and screening tasks.

Abstract

Conformal prediction (CP) is an important tool for distribution-free predictive uncertainty quantification. Yet, a major challenge is to balance computational efficiency and prediction accuracy, particularly for multiple predictions. We propose Leave-One-Out Stable Conformal Prediction (LOO-StabCP), a novel method to speed up full conformal using algorithmic stability without sample splitting. By leveraging leave-one-out stability, our method is much faster in handling a large number of prediction requests compared to existing method RO-StabCP based on replace-one stability. We derived stability bounds for several popular machine learning tools: regularized loss minimization (RLM) and stochastic gradient descent (SGD), as well as kernel method, neural networks and bagging. Our method is theoretically justified and demonstrates superior numerical performance on synthetic and real-world data. We applied our method to a screening problem, where its effective exploitation of training data led to improved test power compared to state-of-the-art method based on split conformal.

Leave-One-Out Stable Conformal Prediction

TL;DR

This work addresses the computational bottleneck of full conformal prediction when multiple predictions are required by introducing Leave-One-Out Stable Conformal Prediction (LOO-StabCP), which uses leave-one-out stability to correct a single model fit on the training data. The method preserves finite-sample coverage while dramatically reducing the need to refit models for each test point, outperforming RO-StabCP in large-scale prediction tasks. The authors derive LOO stability bounds for Regularized Loss Minimization (RLM) and Stochastic Gradient Descent (SGD), extend the framework to kernel methods, neural networks, and bagging, and validate the approach through simulations and real data, including a conformalized screening application (LOO-cfBH) that improves power under FDR control. Overall, LOO-StabCP enables scalable, distribution-free uncertainty quantification with strong practical impact for large-scale prediction and screening tasks.

Abstract

Conformal prediction (CP) is an important tool for distribution-free predictive uncertainty quantification. Yet, a major challenge is to balance computational efficiency and prediction accuracy, particularly for multiple predictions. We propose Leave-One-Out Stable Conformal Prediction (LOO-StabCP), a novel method to speed up full conformal using algorithmic stability without sample splitting. By leveraging leave-one-out stability, our method is much faster in handling a large number of prediction requests compared to existing method RO-StabCP based on replace-one stability. We derived stability bounds for several popular machine learning tools: regularized loss minimization (RLM) and stochastic gradient descent (SGD), as well as kernel method, neural networks and bagging. Our method is theoretically justified and demonstrates superior numerical performance on synthetic and real-world data. We applied our method to a screening problem, where its effective exploitation of training data led to improved test power compared to state-of-the-art method based on split conformal.

Paper Structure

This paper contains 41 sections, 8 theorems, 54 equations, 8 figures, 4 tables, 9 algorithms.

Table of Contents

  1. Introduction
  2. Prior Works on Conformal Prediction (CP)
  3. LOO-StabCP: Leave-One-Out Stable Conformal Prediction
  4. Leave-one-out (LOO) stability and general framework
  5. LOO Stable Algorithms
  6. Example 1: Regularized Loss Minimization (RLM)
  7. Example 2: Stochastic Gradient Descent (SGD)
  8. Towards Broader Applicability of LOO-StabCP
  9. Kernel method:
  10. Neural networks:
  11. Bagging:
  12. Simulation
  13. Data Examples
  14. Application: Conformalized Screening
  15. Conclusion and Future Work
  16. ...and 26 more sections

Key Result

Theorem 1

If the prediction method is leave-one-out stable as in Definition definition::LOO-stability. Then for each $j\in [m]$, the prediction set $\mathcal{C}^{\mathrm{LOO}}_{j,\alpha}(X_{n+j})$ constructed by Algorithm alg:loo_stab satisfies

Figures (8)

  • Figure 1: Comparison of CP methods. Our method ( LOO-StabCP) achieves competitive prediction accuracy and computes at the speed comparable to SplitCP, while maintaining coverage validity.
  • Figure 2: Comparison of prediction interval lengths, under choices of $m=1$ and $m=100$.
  • Figure 3: Comparison of CP methods with neural networks with single hidden layer under choice of $m=100$. LOO-StabCP continues to closely achieve the target coverage while exhibiting lower variability in prediction intervals.
  • Figure 4: Comparison of screening methods on recruitment data. Time cost does not vary with $q$.
  • Figure 5: Comparison of CP methods using kernelized technique.
  • ...and 3 more figures

Theorems & Definitions (18)

  • Definition 1: Replace-One Algorithmic Stability
  • Definition 2: Leave-One-Out Algorithmic Stability
  • Theorem 1
  • Definition 3: $\rho$-Lipschitz
  • Definition 4: Strong Convexity
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • ...and 8 more