Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift

Farbod Siahkali, Ashwin Verma, Vijay Gupta

TL;DR

This paper addresses the breakdown of conformal prediction guarantees under distribution shift when target labels are unavailable. It leverages domain-adaptation insights to bound target coverage in terms of the source classifier loss and a Wasserstein shift, deriving a concrete lower bound that includes $L_r(f,P)$ and a shift term, and introduces relaxed pseudo-calibrated sets with a slack parameter to guarantee prescribed target coverage. A novel source-tuned pseudo-calibration algorithm is proposed, interpolating between hard pseudo-labels and randomized labels based on an uncertainty measure to reduce conservatism while preserving coverage. Numerical experiments on MNIST and CIFAR datasets show that the bounds qualitatively track pseudo-calibration behavior and that the proposed method mitigates coverage degradation under distribution shift with reasonable prediction-set sizes. Overall, the work provides theory-backed, practical tools for reliable multiclass prediction under shift in the absence of target labels.

Abstract

Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.

Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift

TL;DR

This paper addresses the breakdown of conformal prediction guarantees under distribution shift when target labels are unavailable. It leverages domain-adaptation insights to bound target coverage in terms of the source classifier loss and a Wasserstein shift, deriving a concrete lower bound that includes and a shift term, and introduces relaxed pseudo-calibrated sets with a slack parameter to guarantee prescribed target coverage. A novel source-tuned pseudo-calibration algorithm is proposed, interpolating between hard pseudo-labels and randomized labels based on an uncertainty measure to reduce conservatism while preserving coverage. Numerical experiments on MNIST and CIFAR datasets show that the bounds qualitatively track pseudo-calibration behavior and that the proposed method mitigates coverage degradation under distribution shift with reasonable prediction-set sizes. Overall, the work provides theory-backed, practical tools for reliable multiclass prediction under shift in the absence of target labels.

Abstract

Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.
Paper Structure (15 sections, 5 theorems, 26 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 15 sections, 5 theorems, 26 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Problem Formulation
  3. Conformal Prediction
  4. Distribution shift measure
  5. Problem Considered
  6. Analytical Results
  7. Coverage Gap Upper Bounds under Distribution Shift
  8. Source-Tuned Pseudo-Calibration
  9. Numerical Experiments
  10. Conclusion
  11. Proof of Lemma \ref{['lemma:W1upperbounds']}
  12. Kantorovich-Rubinstein Inequality
  13. Proof of Theorem \ref{['thm:pseudo-calibration']}
  14. Proof of Corollary \ref{['cor:hinge_tau']}
  15. Proof of Theorem \ref{['thm:relative_coverage']}

Key Result

Lemma 1

Under Assumptions assump:boundlip and assump:shift, we have

Figures (2)

  • Figure 1: Coverage on $Q_\sigma$ vs. shift $\sigma$ for source calibration, hard pseudo-calibration, and source-tuned pseudo-calibration. Dashed curves show the coverage lower bounds from Theorem \ref{['thm:pseudo-calibration']}.
  • Figure 2: Coverage (solid, left axis) and ESS (dashed, right axis) on $Q_\sigma$ vs. shift. Colors denote the method: hard pseudo-calibration $\tilde{q}_{Q_\sigma,\alpha}$, $\tau$-adjusted $\tilde{q}_{Q_\sigma,\alpha}+\tau(\sigma)$, and oracle $q_{Q_\sigma,\alpha}$. The black curve is the hinge-loss lower bound. Increasing $\tau$ maintains the coverage lower bound at the cost of larger ESS.

Theorems & Definitions (6)

  • Definition 1
  • Lemma 1
  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Lemma 2