Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Conformal Prediction under Data Heterogeneity

Vincent Plassier, Nikita Kotelevskii, Aleksandr Rubashevskii, Fedor Noskov, Maksim Velikanov, Alexander Fishkov, Samuel Horvath, Martin Takac, Eric Moulines, Maxim Panov

TL;DR

This work introduces a new efficient approach to CP that produces provably valid confidence sets for fairly general non-exchangeable data distributions and illustrates the general theory with applications to the challenging setting of federated learning under data heterogeneity between agents.

Abstract

Conformal Prediction (CP) stands out as a robust framework for uncertainty quantification, which is crucial for ensuring the reliability of predictions. However, common CP methods heavily rely on data exchangeability, a condition often violated in practice. Existing approaches for tackling non-exchangeability lead to methods that are not computable beyond the simplest examples. This work introduces a new efficient approach to CP that produces provably valid confidence sets for fairly general non-exchangeable data distributions. We illustrate the general theory with applications to the challenging setting of federated learning under data heterogeneity between agents. Our method allows constructing provably valid personalized prediction sets for agents in a fully federated way. The effectiveness of the proposed method is demonstrated in a series of experiments on real-world datasets.

Efficient Conformal Prediction under Data Heterogeneity

TL;DR

This work introduces a new efficient approach to CP that produces provably valid confidence sets for fairly general non-exchangeable data distributions and illustrates the general theory with applications to the challenging setting of federated learning under data heterogeneity between agents.

Abstract

Conformal Prediction (CP) stands out as a robust framework for uncertainty quantification, which is crucial for ensuring the reliability of predictions. However, common CP methods heavily rely on data exchangeability, a condition often violated in practice. Existing approaches for tackling non-exchangeability lead to methods that are not computable beyond the simplest examples. This work introduces a new efficient approach to CP that produces provably valid confidence sets for fairly general non-exchangeable data distributions. We illustrate the general theory with applications to the challenging setting of federated learning under data heterogeneity between agents. Our method allows constructing provably valid personalized prediction sets for agents in a fully federated way. The effectiveness of the proposed method is demonstrated in a series of experiments on real-world datasets.
Paper Structure (30 sections, 13 theorems, 152 equations, 8 figures, 1 table, 1 algorithm)

This paper contains 30 sections, 13 theorems, 152 equations, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Conformal Prediction under Data Heterogeneity
  3. Basics of Conformal Prediction
  4. CP applications beyond exchangeable cases.
  5. Conformal prediction under covariate shift.
  6. Efficient Conformal Prediction under Data Shift
  7. Theoretical guarantees.
  8. Sketch of Proof.
  9. Federated Conformal Prediction under Covariate Shift
  10. Federated inference procedure.
  11. Federated ratios estimation.
  12. Federated quantile estimation.
  13. Related Work
  14. Experiments
  15. Domain Adaptation on Synthetic Data.
  16. ...and 15 more sections

Key Result

Theorem 2.1

Assume there are no ties between $\{V_k\}_{k\in[N+1]}\cup\{\infty\}$ almost surely. If $\{\lambda_{k}-\mathbb{E}\lambda_k\}_{k \in [N]}$ are sub-Gaussian random variables with parameters $\sigma_{1}, \ldots, \sigma_{N}$, then, for any $\delta \in (0,1)$, with probability at least $1 - \delta$, it ho where we denote

Figures (8)

  • Figure 1: Example of synthetic data. We show the true dependence as a dotted line, neural network prediction and the predictive confidence interval are in red. Additionally, we present PDFs of the train and test distributions on a secondary vertical axis.
  • Figure 2: CIFAR-10 experimental results: (a) The distribution of coverage percentage for each agent. It shows how closely the predicted values covers the true values across varying degrees of data corruption. (b) The distribution of set sizes for different agents. The plot illustrates the growth in set sizes as data corruption increases, emphasizing the relationship between data integrity and set size.
  • Figure 3: CIFAR-100 experimental results: (a) Mean empirical coverage changes as function of the calibration dataset size. (b) Standard deviation of empirical coverage as function of the calibration dataset size.
  • Figure 4: ImageNet experimental results: (a) Empirical coverage of conformal prediction sets as a function of data corruption level. It shows how accurately do conformal sets capture the true classes of the data. (b) Average set size of conformal prediction sets as a function of data corruption level. The size of the sets increases with the level of corruption due to the increasing uncertainty of the model based on the corrupted data.
  • Figure 5: ImageNet & ImageNet-R experimental results: (a) Empirical coverage of conformal prediction sets for non-corrupted and corrupted data. It shows how accurately do conformal sets capture the true classes of the data. (b) Average set size of conformal prediction sets for non-corrupted and corrupted data. The size of the sets increases with the level of corruption due to the increasing uncertainty of the model based on the corrupted data.
  • ...and 3 more figures

Theorems & Definitions (24)

  • Theorem 2.1
  • Theorem 2.2
  • Lemma 1.1
  • proof
  • Theorem 1.2
  • proof
  • Lemma 1.3
  • proof
  • Lemma 1.4
  • proof
  • ...and 14 more