The Window Dilemma: Why Concept Drift Detection is Ill-Posed

TL;DR

This paper argues that concept drift detection is fundamentally ill-posed and heavily dependent on how data are windowed, a phenomenon they term the Window Dilemma. Through both theoretical framing and experiments across real-world streams, the authors show that drift-aware detectors often do not outperform drift-unaware or batch retraining strategies, calling into question the practical value of binary drift detection. The work advocates shifting focus from detecting when drift occurs to understanding where and how drift manifests, and integrating drift insights with broader learning strategies such as transfer learning and change mining. Overall, the findings suggest that choosing a robust classifier and using periodic retraining can outperform many drift-detection approaches, prompting a reevaluation of current drift-detection paradigms in data streams.

Abstract

Non-stationarity of an underlying data generating process that leads to distributional changes over time is a key characteristic of Data Streams. This phenomenon, commonly referred to as Concept Drift, has been intensively studied, and Concept Drift Detectors have been established as a class of methods for detecting such changes (drifts). For the most part, Drift Detectors compare regions (windows) of the data stream and detect drift if those windows are sufficiently dissimilar. In this work, we introduce the Window Dilemma, an observation that perceived drift is a product of windowing and not necessarily the underlying data generating process. Additionally, we highlight that drift detection is ill-posed, primarily because verification of drift events are implausible in practice. We demonstrate these contributions first by an illustrative example, followed by empirical comparisons of drift detectors against a variety of alternative adaptation strategies. Our main finding is that traditional batch learning techniques often perform better than their drift-aware counterparts further bringing into question the purpose of detectors in Stream Classification.

Figures (1)

• Figure 1: An illustrative example of the Window Dilemma. In the top left we have two distributions $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ for which we would like to detect Concept Drift. In practice, we have to achieve this by observing samples generated over the timesteps $i=0$ and $j=100$ and keeping them in a Window $W(i,j)$. By choosing some intermediate point $k$, we can construct two sub-windows which we would use to detect drift. The Window Dilemma shows that, depending on our choice of $k$, we will observe varying distributions, none of which are the distributions we are actually trying to compare. In this Incremental Drift setting, choosing a $k$ close to either $i$ or $j$ will approximate the generating distributions of either $\mathcal{D}_{i}$ (top right figure) or $\mathcal{D}_{j}$ (bottom right figure) accurately, at the cost of poorly approximating the other. In the case where $k$ is chosen as the mid-point (bottom left figure), as often done in practice, we see that we get the right shape for each distribution, but the overlap between them isn't representative of the overlap between the actual distributions $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$.