Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OPTWIN: Drift identification with optimal sub-windows

Mauro Dalle Lucca Tosi, Martin Theobald

TL;DR

OPTWIN addresses the challenge of concept drift in online learning by introducing an optimal-window drift detector that splits a sliding window of errors at the most informative point using both $t$-tests (means) and $f$-tests (variances). The method provides theoretical false-positive and false-negative guarantees and delivers practical benefits, including higher F1-scores and up to a 21% reduction in retraining time compared to strong baselines like ADWIN. Across MOA-based experiments, synthetic and real datasets, and neural-network drift scenarios, OPTWIN achieves superior precision and recall with drift delays comparable to or better than existing detectors. Its implementation combines precomputed statistical quantiles with constant-time per-element updates, enabling scalable deployment in real-time OL systems. The work demonstrates that incorporating variance changes alongside mean shifts yields more reliable drift detection in diverse streaming contexts.

Abstract

Online Learning (OL) is a field of research that is increasingly gaining attention both in academia and industry. One of the main challenges of OL is the inherent presence of concept drifts, which are commonly defined as unforeseeable changes in the statistical properties of an incoming data stream over time. The detection of concept drifts typically involves analyzing the error rates produced by an underlying OL algorithm in order to identify if a concept drift occurred or not, such that the OL algorithm can adapt accordingly. Current concept-drift detectors perform very well, i.e., with low false negative rates, but they still tend to exhibit high false positive rates in the concept-drift detection. This may impact the performance of the learner and result in an undue amount of computational resources spent on retraining a model that actually still performs within its expected range. In this paper, we propose OPTWIN, our "OPTimal WINdow" concept drift detector. OPTWIN uses a sliding window of events over an incoming data stream to track the errors of an OL algorithm. The novelty of OPTWIN is to consider both the means and the variances of the error rates produced by a learner in order to split the sliding window into two provably optimal sub-windows, such that the split occurs at the earliest event at which a statistically significant difference according to either the $t$- or the $f$-tests occurred. We assessed OPTWIN over the MOA framework, using ADWIN, DDM, EDDM, STEPD and ECDD as baselines over 7 synthetic and real-world datasets, and in the presence of both sudden and gradual concept drifts. In our experiments, we show that OPTWIN surpasses the F1-score of the baselines in a statistically significant manner while maintaining a lower detection delay and saving up to 21% of time spent on retraining the models.

OPTWIN: Drift identification with optimal sub-windows

TL;DR

OPTWIN addresses the challenge of concept drift in online learning by introducing an optimal-window drift detector that splits a sliding window of errors at the most informative point using both -tests (means) and -tests (variances). The method provides theoretical false-positive and false-negative guarantees and delivers practical benefits, including higher F1-scores and up to a 21% reduction in retraining time compared to strong baselines like ADWIN. Across MOA-based experiments, synthetic and real datasets, and neural-network drift scenarios, OPTWIN achieves superior precision and recall with drift delays comparable to or better than existing detectors. Its implementation combines precomputed statistical quantiles with constant-time per-element updates, enabling scalable deployment in real-time OL systems. The work demonstrates that incorporating variance changes alongside mean shifts yields more reliable drift detection in diverse streaming contexts.

Abstract

Online Learning (OL) is a field of research that is increasingly gaining attention both in academia and industry. One of the main challenges of OL is the inherent presence of concept drifts, which are commonly defined as unforeseeable changes in the statistical properties of an incoming data stream over time. The detection of concept drifts typically involves analyzing the error rates produced by an underlying OL algorithm in order to identify if a concept drift occurred or not, such that the OL algorithm can adapt accordingly. Current concept-drift detectors perform very well, i.e., with low false negative rates, but they still tend to exhibit high false positive rates in the concept-drift detection. This may impact the performance of the learner and result in an undue amount of computational resources spent on retraining a model that actually still performs within its expected range. In this paper, we propose OPTWIN, our "OPTimal WINdow" concept drift detector. OPTWIN uses a sliding window of events over an incoming data stream to track the errors of an OL algorithm. The novelty of OPTWIN is to consider both the means and the variances of the error rates produced by a learner in order to split the sliding window into two provably optimal sub-windows, such that the split occurs at the earliest event at which a statistically significant difference according to either the - or the -tests occurred. We assessed OPTWIN over the MOA framework, using ADWIN, DDM, EDDM, STEPD and ECDD as baselines over 7 synthetic and real-world datasets, and in the presence of both sudden and gradual concept drifts. In our experiments, we show that OPTWIN surpasses the F1-score of the baselines in a statistically significant manner while maintaining a lower detection delay and saving up to 21% of time spent on retraining the models.
Paper Structure (16 sections, 1 theorem, 4 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 16 sections, 1 theorem, 4 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Works
  3. Background
  4. Related Works
  5. OPTWIN -- Optimal Window Concept Drift Detector
  6. Definition of concepts
  7. Setup & Parameters
  8. Assumptions
  9. Algorithm
  10. Proof of Theorem \ref{['theorem']}
  11. Implementation & Analysis
  12. Experiments & Results
  13. MOA Experiments
  14. Classification Experiments
  15. Neural Network Experiments
  16. ...and 1 more sections

Key Result

Theorem 3.1

Figures (5)

  • Figure 1: Concept drift types.
  • Figure 2: Sudden binary drift detection with average FP rates compared to drift-detection delays.
  • Figure 3: Gradual binary drift detection with average TP and FP rates compared to drift-detection delays.
  • Figure 4: TP and FP rates for drift detection on the AGRAWAL dataset with sudden concept drifts.
  • Figure 5: Sudden drift detection over the loss of a NN.

Theorems & Definitions (1)

  • Theorem 3.1