Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-model Ensemble Conformal Prediction in Dynamic Environments

Erfan Hajihashemi, Yanning Shen

TL;DR

This work introduces a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models, and is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage.

Abstract

Conformal prediction is an uncertainty quantification method that constructs a prediction set for a previously unseen datum, ensuring the true label is included with a predetermined coverage probability. Adaptive conformal prediction has been developed to address data distribution shifts in dynamic environments. However, the efficiency of prediction sets varies depending on the learning model used. Employing a single fixed model may not consistently offer the best performance in dynamic environments with unknown data distribution shifts. To address this issue, we introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage. Experiments on real and synthetic datasets corroborate that the proposed approach consistently yields more efficient prediction sets while maintaining valid coverage, outperforming alternative methods.

Multi-model Ensemble Conformal Prediction in Dynamic Environments

TL;DR

This work introduces a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models, and is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage.

Abstract

Conformal prediction is an uncertainty quantification method that constructs a prediction set for a previously unseen datum, ensuring the true label is included with a predetermined coverage probability. Adaptive conformal prediction has been developed to address data distribution shifts in dynamic environments. However, the efficiency of prediction sets varies depending on the learning model used. Employing a single fixed model may not consistently offer the best performance in dynamic environments with unknown data distribution shifts. To address this issue, we introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage. Experiments on real and synthetic datasets corroborate that the proposed approach consistently yields more efficient prediction sets while maintaining valid coverage, outperforming alternative methods.

Paper Structure

This paper contains 18 sections, 7 theorems, 57 equations, 2 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Methodology
  4. Multi-model Conformal Prediction in Static Environments
  5. Multi-model Ensemble Conformal Prediction In Dynamic Environments
  6. Experiments
  7. Results
  8. Proofs
  9. Proof of lemma \ref{['lem:bounded-alpha']}
  10. Proof of Theorem \ref{['thm:regret-alg1']}, Regret for MOCP:
  11. Proof of Theorem \ref{['thm:coverage-alg2']}, Coverage error for SAMOCP:
  12. Proof of Theorem \ref{['thm:regret-alg2']}, static regret for SAMOCP
  13. Proof of Lemma \ref{['lem:dynamic-regret-samocp']}, Dynamic regret for SAMOCP:
  14. Additional Experiments
  15. Synthetic Dataset
  16. ...and 3 more sections

Key Result

Theorem 1

Algorithm alg:alg1-model-selection achieves the following regret bound in a static environment

Figures (2)

  • Figure 1: Expert creation over 5 time steps using lifetime formula \ref{['eq:lifetime']} when $g = 1$. At each time $t$, an expert is created, marked by a filled circle to indicate the start of the activity, and an unfilled circle to denote the end of the expert's activity.
  • Figure 2: Evaluation of average regret over different interval sizes ($50, 100, \ldots, 500$). Note that for previous methods relying on a single model, the lowest regret across the $4$ learning models is selected.

Theorems & Definitions (7)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4