Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Predictive Systems Under Covariate Shift

Jef Jonkers, Glenn Van Wallendael, Luc Duchateau, Sofie Van Hoecke

TL;DR

This paper proposes Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions, and demonstrates its utility through empirical evaluations on both synthetic and real-world datasets.

Abstract

Conformal Predictive Systems (CPS) offer a versatile framework for constructing predictive distributions, allowing for calibrated inference and informative decision-making. However, their applicability has been limited to scenarios adhering to the Independent and Identically Distributed (IID) model assumption. This paper extends CPS to accommodate scenarios characterized by covariate shifts. We therefore propose Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions. This extension enables the construction of nonparametric predictive distributions capable of handling covariate shifts. We present theoretical underpinnings and conjectures regarding the validity and efficacy of WCPS and demonstrate its utility through empirical evaluations on both synthetic and real-world datasets. Our simulation experiments indicate that WCPS are probabilistically calibrated under covariate shift.

Conformal Predictive Systems Under Covariate Shift

TL;DR

This paper proposes Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions, and demonstrates its utility through empirical evaluations on both synthetic and real-world datasets.

Abstract

Conformal Predictive Systems (CPS) offer a versatile framework for constructing predictive distributions, allowing for calibrated inference and informative decision-making. However, their applicability has been limited to scenarios adhering to the Independent and Identically Distributed (IID) model assumption. This paper extends CPS to accommodate scenarios characterized by covariate shifts. We therefore propose Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions. This extension enables the construction of nonparametric predictive distributions capable of handling covariate shifts. We present theoretical underpinnings and conjectures regarding the validity and efficacy of WCPS and demonstrate its utility through empirical evaluations on both synthetic and real-world datasets. Our simulation experiments indicate that WCPS are probabilistically calibrated under covariate shift.
Paper Structure (20 sections, 19 equations, 4 figures, 1 table)

This paper contains 20 sections, 19 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Conformal Prediction
  4. Covariate Shift
  5. Weighted Conformal Prediction
  6. Conformal Predictive Systems
  7. Split Conformal Predictive Systems
  8. Weighted Conformal Predictive System
  9. Weighted Split Conformal Predictive Systems
  10. Experiments
  11. Data
  12. Empirical Data
  13. Synthetic Data
  14. Results
  15. Coverage of intervals under covariate shift
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Empirical coverage of 80% prediction intervals from (W)SCPS, computed using 1000 different random splits of the airfoil and synthetic dataset.
  • Figure 2: Empirical CRPS of (W)SCPS, computed using 1000 different experiment trials for both airfoil and synthetic datasets.
  • Figure 3: Post-hoc Friedman-Nemenyi test for CRPS.
  • Figure 4: Distribution of p-values of SCPS under IID model (blue), covariate shift (orange), and WSCPS (green). The red dashed line represents the uniform distribution the p-values need to follow so that the (W)SCPS is probabilistically calibrated.

Theorems & Definitions (6)

  • definition 1: Conformal Transducer, vovk_algorithmic_2022
  • definition 2: RPS, vovk_nonparametric_2019
  • definition 3: Inductive (Split) Conformity Measure, vovk_algorithmic_2022
  • definition 4: Split Conformal Transducer, vovk_computationally_2020
  • definition 5: Isotonic Split Conformity Measure, vovk_computationally_2020
  • definition 6: Balanced Isotonic Split Conformity Measure, vovk_computationally_2020