Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Few Observations on Sample-Conditional Coverage in Conformal Prediction

John C. Duchi

TL;DR

This work assesses conditional validity within conformal prediction, demonstrating that split-conformal methods can provide high-probability sample-conditional guarantees and extending to approximate weighted-conditional coverage via quantile-regression on held-out data. It formalizes a weighted-conditional framework using a function class W and shows minimax-rate optimal guarantees for approximate conditional coverage, supported by empirical process-based bounds. The paper develops sharp, rate-optimal bounds leveraging Talagrand-type concentration and VC-dimension tools, and provides building-block proofs for one- and two-sided coverage deviations in various settings, including distinct-score scenarios. Through synthetic experiments and CIFAR-100 experiments, it confirms the practical viability of adaptive-threshold split-conformal predictions, while also highlighting computational advantages and areas needing further development to achieve exact conditional validity in high-dimensional, data-limited regimes.

Abstract

We revisit the problem of constructing predictive confidence sets for which we wish to obtain some type of conditional validity. We provide new arguments showing how ``split conformal'' methods achieve near desired coverage levels with high probability, a guarantee conditional on the validation data rather than marginal over it. In addition, we directly consider (approximate) conditional coverage, where, e.g., conditional on a covariate $X$ belonging to some group of interest, we would like a guarantee that a predictive set covers the true outcome $Y$. We show that the natural method of performing quantile regression on a held-out (validation) dataset yields minimax optimal guarantees of coverage here. Complementing these positive results, we also provide experimental evidence that interesting work remains to be done to develop computationally efficient but valid predictive inference methods.

A Few Observations on Sample-Conditional Coverage in Conformal Prediction

TL;DR

This work assesses conditional validity within conformal prediction, demonstrating that split-conformal methods can provide high-probability sample-conditional guarantees and extending to approximate weighted-conditional coverage via quantile-regression on held-out data. It formalizes a weighted-conditional framework using a function class W and shows minimax-rate optimal guarantees for approximate conditional coverage, supported by empirical process-based bounds. The paper develops sharp, rate-optimal bounds leveraging Talagrand-type concentration and VC-dimension tools, and provides building-block proofs for one- and two-sided coverage deviations in various settings, including distinct-score scenarios. Through synthetic experiments and CIFAR-100 experiments, it confirms the practical viability of adaptive-threshold split-conformal predictions, while also highlighting computational advantages and areas needing further development to achieve exact conditional validity in high-dimensional, data-limited regimes.

Abstract

We revisit the problem of constructing predictive confidence sets for which we wish to obtain some type of conditional validity. We provide new arguments showing how ``split conformal'' methods achieve near desired coverage levels with high probability, a guarantee conditional on the validation data rather than marginal over it. In addition, we directly consider (approximate) conditional coverage, where, e.g., conditional on a covariate belonging to some group of interest, we would like a guarantee that a predictive set covers the true outcome . We show that the natural method of performing quantile regression on a held-out (validation) dataset yields minimax optimal guarantees of coverage here. Complementing these positive results, we also provide experimental evidence that interesting work remains to be done to develop computationally efficient but valid predictive inference methods.

Paper Structure

This paper contains 23 sections, 21 theorems, 156 equations, 3 figures.

Table of Contents

  1. Introduction and background
  2. On $X$-conditional coverage
  3. Sample-conditional coverage
  4. Sample conditional coverage revisited
  5. An elementary proof of Proposition \ref{['proposition:sample-conditional-basic']}
  6. A proof of Proposition \ref{['proposition:sample-conditional-basic']} using uniform convergence
  7. Proof of Proposition \ref{['proposition:distinct-scores']}
  8. Approaching conditional coverage
  9. An estimated confidence set
  10. Proof of Theorem \ref{['theorem:approximate-conditional-coverage']}
  11. Sharper and rate-optimal approximate conditional bounds
  12. Proof of Theorems \ref{['theorem:one-sided-sharp']} and \ref{['theorem:two-sided-sharp']}: building blocks
  13. Proof of Theorem \ref{['theorem:one-sided-sharp']}: nonnegative weights
  14. Proof of Theorem \ref{['theorem:two-sided-sharp']}: distinct scores
  15. Experimental Results
  16. ...and 8 more sections

Key Result

Corollary 1.1

Let $\mathcal{X}$ be a metric space and assume that $X \in \mathcal{X}$ has continuous distribution. If $\widehat{C}$ provides distribution free $(1 - \alpha)$ conditional coverage, then for $P$-almost all $x \in \mathcal{X}$,

Figures (3)

  • Figure 1: Impact of the correction to $\alpha$ used in fitting the conformal predictor \ref{['eqn:empirical-quantile-estimator']} for a desired level $\alpha_{\textup{des}} = .1$, i.e., 90% coverage. The "None" correction uses $\alpha = \alpha_{\textup{des}}$, "Naive" uses the correction \ref{['eqn:naive-alpha']}, and "Scaling" uses the correction \ref{['eqn:bai-alpha']}. (a) Coverage rates with the desired coverage marked as the red line. (b) Width of predictive intervals $\widehat{C}(x) = \{y \in \mathbb{R} \mid |\widehat{f}(x) - y| \le \widehat{\theta}^\top \phi(x)\}$.
  • Figure 2: Comparison of full- and split-conformal methods on the simulated sinusoidal data of Sec. \ref{['sec:offline-sin-simulation']} with $n_{\textup{train}} = 200$ training examples and target miscoverage $\alpha = .1$. Plots (a) and (c) use validation sample sizes $n_{\textup{val}} = 20 k = 100$, while (b) and (d) use $n_{\textup{val}} = 160 k = 800$. Plots (a) and (b) show miscoverage $\mathbb{P}(Y \not \in \widehat{C}(X) \mid X \in B_i)$ by group $B_i$; plots (c) and (d) prediction interval lengths.
  • Figure 3: Coverage of full conformal, split conformal, and static split conformal methods on random 20% "slices" of CIFAR-100 data.

Theorems & Definitions (22)

  • Corollary 1.1
  • Definition 1.1
  • Proposition 1: Vovk Vovk12, Proposition 2
  • Corollary 2.1
  • Proposition 2
  • Lemma 2.1: Bounded differences
  • Corollary 3.1
  • Theorem 1
  • Corollary 3.2
  • Corollary 3.3
  • ...and 12 more