Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalization and Informativeness of Weighted Conformal Risk Control Under Covariate Shift

Matteo Zecchin, Fredrik Hellström, Sangwoo Park, Shlomo Shamai, Osvaldo Simeone

TL;DR

This work analyzes the generalization and efficiency of Weighted Conformal Risk Control (W-CRC) under covariate shift, linking the average size of predictive sets to the base predictor's generalization gap, the covariate-shift magnitude, and data-splitting hyperparameters. A novel information-theoretic bound is derived, showing how calibration size, test-time likelihood ratios, and training-calibration trade-offs influence set informativeness. The results provide practical guidance for allocating data between training and calibration, especially under larger shifts, and are validated on fingerprinting-based localization with RSSI features. Overall, the paper advances understanding of when and how W-CRC remains informative while maintaining reliability under distribution changes.

Abstract

Predictive models are often required to produce reliable predictions under statistical conditions that are not matched to the training data. A common type of training-testing mismatch is covariate shift, where the conditional distribution of the target variable given the input features remains fixed, while the marginal distribution of the inputs changes. Weighted conformal risk control (W-CRC) uses data collected during the training phase to convert point predictions into prediction sets with valid risk guarantees at test time despite the presence of a covariate shift. However, while W-CRC provides statistical reliability, its efficiency -- measured by the size of the prediction sets -- can only be assessed at test time. In this work, we relate the generalization properties of the base predictor to the efficiency of W-CRC under covariate shifts. Specifically, we derive a bound on the inefficiency of the W-CRC predictor that depends on algorithmic hyperparameters and task-specific quantities available at training time. This bound offers insights on relationships between the informativeness of the prediction sets, the extent of the covariate shift, and the size of the calibration and training sets. Experiments on fingerprinting-based localization validate the theoretical results.

Generalization and Informativeness of Weighted Conformal Risk Control Under Covariate Shift

TL;DR

This work analyzes the generalization and efficiency of Weighted Conformal Risk Control (W-CRC) under covariate shift, linking the average size of predictive sets to the base predictor's generalization gap, the covariate-shift magnitude, and data-splitting hyperparameters. A novel information-theoretic bound is derived, showing how calibration size, test-time likelihood ratios, and training-calibration trade-offs influence set informativeness. The results provide practical guidance for allocating data between training and calibration, especially under larger shifts, and are validated on fingerprinting-based localization with RSSI features. Overall, the paper advances understanding of when and how W-CRC remains informative while maintaining reliability under distribution changes.

Abstract

Predictive models are often required to produce reliable predictions under statistical conditions that are not matched to the training data. A common type of training-testing mismatch is covariate shift, where the conditional distribution of the target variable given the input features remains fixed, while the marginal distribution of the inputs changes. Weighted conformal risk control (W-CRC) uses data collected during the training phase to convert point predictions into prediction sets with valid risk guarantees at test time despite the presence of a covariate shift. However, while W-CRC provides statistical reliability, its efficiency -- measured by the size of the prediction sets -- can only be assessed at test time. In this work, we relate the generalization properties of the base predictor to the efficiency of W-CRC under covariate shifts. Specifically, we derive a bound on the inefficiency of the W-CRC predictor that depends on algorithmic hyperparameters and task-specific quantities available at training time. This bound offers insights on relationships between the informativeness of the prediction sets, the extent of the covariate shift, and the size of the calibration and training sets. Experiments on fingerprinting-based localization validate the theoretical results.
Paper Structure (17 sections, 3 theorems, 53 equations, 4 figures)

This paper contains 17 sections, 3 theorems, 53 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Setting
  4. Training
  5. Weighted Conformal Risk Control
  6. Information-Theoretic Inefficiency Bound for W-CRC
  7. Generalization Properties of the Base Predictor
  8. Analysis of the NC Score
  9. Main Result
  10. Discussion
  11. Experiments
  12. Conclusion
  13. Proof of Proposition \ref{['prop:crc_weighted']}
  14. On the Base Predictor Generalization Error
  15. Bounding $\tilde{\Delta}(Q|\mathcal{D}_{ \rm tr })$
  16. ...and 2 more sections

Key Result

Proposition 1

Under Assumption ass:loss_bound_noincr, the W-CRC predictor eq:crc_set_predictor is $\alpha$-reliable, i.e. where the expectation is over both the test and calibration data as well as over the model parameters, with fixed training set $\mathcal{D}_{ \rm tr }$. This result follows in a manner similar to tibshirani2019conformalangelopoulos2022conformal, with the only caveat that one needs to accoun

Figures (4)

  • Figure 1: In fingerprinting-based localization, received signal strength indicators (RSSI) measured by base stations are used to localize mobile devices. The base stations can be dynamically switched off, and their activation rates are depicted by the colored bars. A portion of the collected RSSI measurements is used to train a prediction model, while the rest are used to convert the model's outputs into prediction regions that meet a specified target risk. Calibration is achieved by W-CRC by accounting for covariate shift due to differing activation rates during training and testing through likelihood ratios between the training and testing marginals tibshirani2019conformalangelopoulos2022conformal.
  • Figure 2: Inefficiency bound as a function of the fraction $\kappa$ of data allocated for training under a fixed data budget $n$ while varying the covariate shift parameter $\bar{W}$.
  • Figure 3: Relative inefficiency of W-CRC as a function of the target reliability level $\alpha$ for different calibration set sizes $n_{ \rm{cal} }$ and various covariate shift settings.
  • Figure 4: Relative inefficiency of W-CRC as a function of the fraction of training data $\kappa$ under a fixed data set of size $n$ and various covariate shift settings.

Theorems & Definitions (4)

  • Proposition 1: Proposition 4angelopoulos2022conformal
  • Theorem 1
  • Definition 1: Weighted Exchangeability tibshirani2019conformal
  • Lemma 1