Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training-Conditional Coverage Bounds under Covariate Shift

Mehrdad Pournaderi, Yu Xiang

TL;DR

This work derives PAC-style bounds for the training-conditional miscoverage of conformal prediction methods under covariate shift, addressing a gap where prior results emphasized marginal coverage. By introducing likelihood-ratio weighting and weighted DKW inequalities, the authors obtain explicit tail guarantees for split conformal, and analogous bounds for full conformal and jackknife+ under uniform stability and bi-Lipschitz assumptions. The bounds transparently depend on the shift severity via an upper bound $B$ on the likelihood ratio $dQ/dP$ and on calibration or sample sizes, providing a practical sense of how much data is needed to maintain training-conditional reliability under shift. The results illuminate the trade-offs and robustness considerations when using weighted conformal methods in shifted environments and suggest avenues for estimating likelihood ratios and extending the framework to robustness and SSL settings.

Abstract

Conformal prediction methodology has recently been extended to the covariate shift setting, where the distribution of covariates differs between training and test data. While existing results ensure that the prediction sets from these methods achieve marginal coverage above a nominal level, their coverage rate conditional on the training dataset (referred to as training-conditional coverage) remains unexplored. In this paper, we address this gap by deriving upper bounds on the tail of the training-conditional coverage distribution, offering probably approximately correct (PAC) guarantees for these methods. Our results characterize the reliability of the prediction sets in terms of the severity of distributional changes and the size of the training dataset.

Training-Conditional Coverage Bounds under Covariate Shift

TL;DR

This work derives PAC-style bounds for the training-conditional miscoverage of conformal prediction methods under covariate shift, addressing a gap where prior results emphasized marginal coverage. By introducing likelihood-ratio weighting and weighted DKW inequalities, the authors obtain explicit tail guarantees for split conformal, and analogous bounds for full conformal and jackknife+ under uniform stability and bi-Lipschitz assumptions. The bounds transparently depend on the shift severity via an upper bound on the likelihood ratio and on calibration or sample sizes, providing a practical sense of how much data is needed to maintain training-conditional reliability under shift. The results illuminate the trade-offs and robustness considerations when using weighted conformal methods in shifted environments and suggest avenues for estimating likelihood ratios and extending the framework to robustness and SSL settings.

Abstract

Conformal prediction methodology has recently been extended to the covariate shift setting, where the distribution of covariates differs between training and test data. While existing results ensure that the prediction sets from these methods achieve marginal coverage above a nominal level, their coverage rate conditional on the training dataset (referred to as training-conditional coverage) remains unexplored. In this paper, we address this gap by deriving upper bounds on the tail of the training-conditional coverage distribution, offering probably approximately correct (PAC) guarantees for these methods. Our results characterize the reliability of the prediction sets in terms of the severity of distributional changes and the size of the training dataset.
Paper Structure (22 sections, 19 theorems, 142 equations, 2 figures)

This paper contains 22 sections, 19 theorems, 142 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Full conformal and split conformal
  4. Jackknife+
  5. Conformal prediction under distribution shift
  6. Conditional Coverage Guarantees
  7. Split conformal
  8. Full conformal and jackknife+
  9. Discussion
  10. Conclusion
  11. Technical Lemmas: Weighted DKW Inequalities
  12. Background
  13. Weighted DKW Inequalities
  14. Proof of Theorem \ref{['thm:splt']}
  15. Proof of Remark \ref{['splt_alt']}
  16. ...and 7 more sections

Key Result

Theorem 1

Let $m<n$ denote the size of the calibration data set. Assume that $Q$ is absolutely continuous with respect to $P$ ($Q \ll P$) and $dQ/dP\leq B<\infty$. Then, for all $\delta > 0$, where $C>0$ is a universal constant. The probability is taken with respect to $P$ since each entry in ${\bf D}_n$ follows $P$.

Figures (2)

  • Figure 1: Histograms of the training-conditional coverage rates are presented for four datasets from the UCI Machine Learning Repository: Wine Quality (top left), Abalone (top right), Concrete Compressive Strength (bottom left), and Combined Cycle Power Plant (bottom right). See Appendix \ref{['sim_details']} for details of this simulation study.
  • Figure 2: The likelihood ratio is estimated using kernel density estimator with Gaussian kernel.

Theorems & Definitions (20)

  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Remark 1
  • Theorem 3: Jackknife+ under exchangeability
  • Theorem 4: Jackknife+ under covariate shift
  • Corollary 2: CV+
  • Theorem 5: Full conformal under exchangeability
  • Theorem 6: Full conformal under covariate shift
  • Theorem 7: van1997weak(Theorem 2.14.2)
  • ...and 10 more