Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Drift Localization using Conformal Predictions

Fabian Hinder, Valerie Vaquet, Johannes Brinkrolf, Barbara Hammer

TL;DR

This work considers a fundamentally different approach based on conformal predictions for drift localization, which discusses and shows the shortcomings of common approaches and demonstrates the performance of this approach on state-of-the-art image datasets.

Abstract

Concept drift -- the change of the distribution over time -- poses significant challenges for learning systems and is of central interest for monitoring. Understanding drift is thus paramount, and drift localization -- determining which samples are affected by the drift -- is essential. While several approaches exist, most rely on local testing schemes, which tend to fail in high-dimensional, low-signal settings. In this work, we consider a fundamentally different approach based on conformal predictions. We discuss and show the shortcomings of common approaches and demonstrate the performance of our approach on state-of-the-art image datasets.

Drift Localization using Conformal Predictions

TL;DR

This work considers a fundamentally different approach based on conformal predictions for drift localization, which discusses and shows the shortcomings of common approaches and demonstrates the performance of this approach on state-of-the-art image datasets.

Abstract

Concept drift -- the change of the distribution over time -- poses significant challenges for learning systems and is of central interest for monitoring. Understanding drift is thus paramount, and drift localization -- determining which samples are affected by the drift -- is essential. While several approaches exist, most rely on local testing schemes, which tend to fail in high-dimensional, low-signal settings. In this work, we consider a fundamentally different approach based on conformal predictions. We discuss and show the shortcomings of common approaches and demonstrate the performance of our approach on state-of-the-art image datasets.
Paper Structure (11 sections, 3 figures, 1 algorithm)

This paper contains 11 sections, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Concept Drift and Drift Localization
  4. Localization Methods and Their Shortcomings
  5. Conformal Prediction
  6. Conformal prediction for drift localization
  7. Experiments
  8. Fish-Head Dataset
  9. Effect of Bootstraps
  10. Evaluation of Method
  11. Conclusion and Future Work

Figures (3)

  • Figure 1: Effect of grouping-test/calibration-set sizes. (Decision tree, Fish-head dataset, 500 samples, 98 drifting)
  • Figure 2: Effect of number of bootstraps on ROC-AUC. Figure shows aggregation of 500 runs on FishHead dataset.
  • Figure 3: Experimental results. ROC-AUC (500 runs) for various drift localizes using windows of 250/60 samples with 98/10 drifting samples (in total).