Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Interpretable Subgroup Drifts

Flavio Giobergia, Eliana Pastor, Luca de Alfaro, Elena Baralis

TL;DR

DriftInspector tackles the limitation of global drift monitoring by enabling fine-grained detection of performance drifts within interpretable data subgroups. It identifies frequent subgroups during training, constructs efficient sparse representations to monitor subgroup performance over time, and detects drift using a per-subgroup Welch's t-test on a Bayesianized performance metric $h(X)$. The approach demonstrates superior sensitivity to small subgroups, provides interpretable drift summaries, and maintains strong performance for global drifts, all with favorable computational efficiency. This enables targeted model maintenance and fairness-aware interventions in dynamic environments where subgroup behavior evolves independently of the overall population.

Abstract

The ability to detect and adapt to changes in data distributions is crucial to maintain the accuracy and reliability of machine learning models. Detection is generally approached by observing the drift of model performance from a global point of view. However, drifts occurring in (fine-grained) data subgroups may go unnoticed when monitoring global drift. We take a different perspective, and introduce methods for observing drift at the finer granularity of subgroups. Relevant data subgroups are identified during training and monitored efficiently throughout the model's life. Performance drifts in any subgroup are detected, quantified and characterized so as to provide an interpretable summary of the model behavior over time. Experimental results confirm that our subgroup-level drift analysis identifies drifts that do not show at the (coarser) global dataset level. The proposed approach provides a valuable tool for monitoring model performance in dynamic real-world applications, offering insights into the evolving nature of data and ultimately contributing to more robust and adaptive models.

Detecting Interpretable Subgroup Drifts

TL;DR

DriftInspector tackles the limitation of global drift monitoring by enabling fine-grained detection of performance drifts within interpretable data subgroups. It identifies frequent subgroups during training, constructs efficient sparse representations to monitor subgroup performance over time, and detects drift using a per-subgroup Welch's t-test on a Bayesianized performance metric . The approach demonstrates superior sensitivity to small subgroups, provides interpretable drift summaries, and maintains strong performance for global drifts, all with favorable computational efficiency. This enables targeted model maintenance and fairness-aware interventions in dynamic environments where subgroup behavior evolves independently of the overall population.

Abstract

The ability to detect and adapt to changes in data distributions is crucial to maintain the accuracy and reliability of machine learning models. Detection is generally approached by observing the drift of model performance from a global point of view. However, drifts occurring in (fine-grained) data subgroups may go unnoticed when monitoring global drift. We take a different perspective, and introduce methods for observing drift at the finer granularity of subgroups. Relevant data subgroups are identified during training and monitored efficiently throughout the model's life. Performance drifts in any subgroup are detected, quantified and characterized so as to provide an interpretable summary of the model behavior over time. Experimental results confirm that our subgroup-level drift analysis identifies drifts that do not show at the (coarser) global dataset level. The proposed approach provides a valuable tool for monitoring model performance in dynamic real-world applications, offering insights into the evolving nature of data and ultimately contributing to more robust and adaptive models.
Paper Structure (36 sections, 16 equations, 16 figures, 7 tables)

This paper contains 36 sections, 16 equations, 16 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Drift detection.
  4. Subgroup analysis.
  5. Problem definition
  6. Method
  7. Identification of Monitored Subgroups
  8. Subgroup membership extraction
  9. Subgroup-wise metric computation
  10. Drift detection
  11. Experimental evaluation
  12. Experimental setting
  13. Local drift datasets
  14. Local drift injection
  15. Global drift datasets
  16. ...and 21 more sections

Figures (16)

  • Figure 1: Adult dataset. Drift event affecting the subgroup women under the age of 36, employed in the private sector (approx. 10% of the overall population). The overall accuracy is only marginally affected, whereas the drop in performance for the drifting subgroup is more prominent.
  • Figure 2: Main steps of the DriftInspector process.
  • Figure 3: Drift detection performance as the target subgroup support varies; Adult. DriftInspector achieves satisfactory results even when small portions of the population drift. Each point obtained from up to 50 positive experiments (i.e., with injected drift) and the same number of negative ones.
  • Figure 4: nDCG varying the target subgroup support; Adult.
  • Figure 5: Evolution of the best threshold value for $\mathbf{\tau_t}$ on Adult, when varying the target subgroup support. The threshold value $\mathbf{\tau_t = 5}$ used in the experiments is shown in blue.
  • ...and 11 more figures