Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Synthetic Benchmark to Explore Limitations of Localized Drift Detections

Flavio Giobergia, Eliana Pastor, Luca de Alfaro, Elena Baralis

TL;DR

This paper tackles the limitation of assuming global concept drift by examining localized drift within subpopulations. It introduces the Subgroup Agrawal Drift Dataset, a synthetic benchmark built on the Agrawal generator where a randomly selected subgroup experiences gradual drift defined by $F = s(x) \cdot [Z \cdot f_i(x) + (1 - Z) \cdot f_j(x)] + (1 - s(x)) f_i(x)$ with $p_t = (1 + e^{-4(t-k)/w})^{-1}$, enabling controlled evaluation of local-drift detectors. The authors evaluate four drift detectors—DDM, EDDM, HDDM, and FHDDM—across drifting-subgroup sizes from $1\%$ to $100\%$ and show that detection performance collapses for small subgroups due to high false negatives, while false positives remain relatively stable. The work provides public code for generating the benchmark and highlights the need for detector designs that can explicitly handle localized drift, with implications for fairness and subpopulation-aware monitoring.

Abstract

Concept drift is a common phenomenon in data streams where the statistical properties of the target variable change over time. Traditionally, drift is assumed to occur globally, affecting the entire dataset uniformly. However, this assumption does not always hold true in real-world scenarios where only specific subpopulations within the data may experience drift. This paper explores the concept of localized drift and evaluates the performance of several drift detection techniques in identifying such localized changes. We introduce a synthetic dataset based on the Agrawal generator, where drift is induced in a randomly chosen subgroup. Our experiments demonstrate that commonly adopted drift detection methods may fail to detect drift when it is confined to a small subpopulation. We propose and test various drift detection approaches to quantify their effectiveness in this localized drift scenario. We make the source code for the generation of the synthetic benchmark available at https://github.com/fgiobergia/subgroup-agrawal-drift.

A Synthetic Benchmark to Explore Limitations of Localized Drift Detections

TL;DR

This paper tackles the limitation of assuming global concept drift by examining localized drift within subpopulations. It introduces the Subgroup Agrawal Drift Dataset, a synthetic benchmark built on the Agrawal generator where a randomly selected subgroup experiences gradual drift defined by with , enabling controlled evaluation of local-drift detectors. The authors evaluate four drift detectors—DDM, EDDM, HDDM, and FHDDM—across drifting-subgroup sizes from to and show that detection performance collapses for small subgroups due to high false negatives, while false positives remain relatively stable. The work provides public code for generating the benchmark and highlights the need for detector designs that can explicitly handle localized drift, with implications for fairness and subpopulation-aware monitoring.

Abstract

Concept drift is a common phenomenon in data streams where the statistical properties of the target variable change over time. Traditionally, drift is assumed to occur globally, affecting the entire dataset uniformly. However, this assumption does not always hold true in real-world scenarios where only specific subpopulations within the data may experience drift. This paper explores the concept of localized drift and evaluates the performance of several drift detection techniques in identifying such localized changes. We introduce a synthetic dataset based on the Agrawal generator, where drift is induced in a randomly chosen subgroup. Our experiments demonstrate that commonly adopted drift detection methods may fail to detect drift when it is confined to a small subpopulation. We propose and test various drift detection approaches to quantify their effectiveness in this localized drift scenario. We make the source code for the generation of the synthetic benchmark available at https://github.com/fgiobergia/subgroup-agrawal-drift.
Paper Structure (10 sections, 2 equations, 4 figures, 1 table)

This paper contains 10 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed dataset
  3. Subgroup Agrawal Drift Dataset
  4. Subgroup definition
  5. Examples of generated subgroups
  6. Experimental results
  7. Drift detection techniques.
  8. Dataset.
  9. Results.
  10. Conclusions

Figures (4)

  • Figure 1: Accuracy computed on the overall dataset and on the drifting subgroup (2% of the dataset), throughout a drifting event.
  • Figure 2: Example of the greedy process adopted to randomly generate subgroups on 2 attributes. From top to bottom, the target subgroup is built by iteratively adding randomly generated slices of the attributes, if their inclusion produces a better approximation of the desired support (in the example, 10%). The checkered area at each step represents the size (support) of the current subgroup.
  • Figure 3: Distribution of the absolute difference between target and corresponding obtained subgroup sizes, for 1,000 generated subgroups and various target sizes, tolerance of 0.01 (marked in red), maximum number of iterations set to 1,000.
  • Figure 4: Performance, in terms of $F_1$ score, accuracy, False Positive Rate, and False Negative Rate, of various drift detection techniques on the binary task of detecting the occurrence of drift events. Subgroup Agrawal Drift dataset, perturbation of 25%, various sizes of drifting subgroups.