Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

Zheng Zhou, Xiangfei Zhang, Chongguang Tao, Yuhong Yang

Abstract

Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeability, but may exhibit systematic undercoverage or overcoverage for specific subpopulations. Assessing conditional validity is challenging, as standard stratification methods suffer from the curse of dimensionality. We propose Conformal Prediction Assessment (CPA), a framework that reframes the evaluation of conditional coverage as a supervised learning task by training a reliability estimator that predicts instance-level coverage probabilities. Building on this estimator, we introduce the Conditional Validity Index (CVI), which decomposes reliability into safety (undercoverage risk) and efficiency (overcoverage cost). We establish convergence rates for the reliability estimator and prove the consistency of CVI-based model selection. Extensive experiments on synthetic and real-world datasets demonstrate that CPA effectively diagnoses local failure modes and that CC-Select, our CVI-based model selection algorithm, consistently identifies predictors with superior conditional coverage performance.

Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

Abstract

Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeability, but may exhibit systematic undercoverage or overcoverage for specific subpopulations. Assessing conditional validity is challenging, as standard stratification methods suffer from the curse of dimensionality. We propose Conformal Prediction Assessment (CPA), a framework that reframes the evaluation of conditional coverage as a supervised learning task by training a reliability estimator that predicts instance-level coverage probabilities. Building on this estimator, we introduce the Conditional Validity Index (CVI), which decomposes reliability into safety (undercoverage risk) and efficiency (overcoverage cost). We establish convergence rates for the reliability estimator and prove the consistency of CVI-based model selection. Extensive experiments on synthetic and real-world datasets demonstrate that CPA effectively diagnoses local failure modes and that CC-Select, our CVI-based model selection algorithm, consistently identifies predictors with superior conditional coverage performance.

Paper Structure

This paper contains 83 sections, 5 theorems, 74 equations, 15 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Uncertainty Quantification via Conformal Prediction
  3. Our Contributions
  4. Related Work
  5. Conditional Coverage in Conformal Prediction.
  6. Model Selection in Conformal Prediction.
  7. Methodology
  8. Asymptotic Behavior of the Conditional Coverage Probability
  9. Estimating the Conditional Coverage Probability Function
  10. Assessment of Conditional Coverage
  11. The Conditional Validity Index (CVI) and its Decomposition
  12. Visualizing Reliability: The CVP Curve
  13. Reliability-Driven Model Selection and Deployment
  14. Theoretical Analysis
  15. Convergence of Conditional Coverage Estimates
  16. ...and 68 more sections

Key Result

Theorem 1

Under Assumption assumption:combined, the finite-sample conditional coverage converges to the asymptotic coverage function $\eta(x) := \mathbb{P}(|\mu(X) + \epsilon - \tilde{\mu}(X)| \leq q_{1 - \alpha} \mid X = x)$ in the following sense: for sufficiently large $n_{\text{pred}}$,

Figures (15)

  • Figure 1: Overview of the predictive reliability learning framework. The pipeline consists of four stages: data preparation, conformal model training, reliability estimation via $\hat{\eta}(x)$, and downstream applications including evaluation, model selection, and deployment.
  • Figure 2:
  • Figure 3:
  • Figure 4:
  • Figure 5:
  • ...and 10 more figures

Theorems & Definitions (11)

  • Theorem 1: Uniform Convergence of $\eta_n(x)$
  • Theorem 2: Convergence of $\hat{\eta}_{n_{\textup{eval}}}(x)$
  • Theorem 3: Consistency of CVI
  • Definition : Asymptotically Better
  • Theorem 4: Consistency of Model Selection via CVI
  • proof
  • proof
  • Lemma 1
  • proof : Proof of lemma \ref{['lm::CLT_surrogate_CVI']}
  • proof : Proof of Theorem \ref{['thm::consistency']}
  • ...and 1 more