Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Model Drift Detection

Pranoy Panda, Kancheti Sai Srinivas, Vineeth N Balasubramanian, Gaurav Sinha

TL;DR

Real-world data distributions evolve over time, causing model drift and potential performance degradation. The paper delivers TRIPODD, a theory-grounded, feature-interaction aware hypothesis-testing framework that uses model risk to detect and interpret drift for both classification and regression. It defines feature-sensitive model drift via $\Delta_p^i(h)$ and builds a hypothesis test around risk differences $\mathcal{R}^S_p(h) - \mathcal{R}^{S\cup\{i\}}_p(h)$, with bootstrap-derived thresholds to control error. Empirical results on synthetic and real-world drift datasets show TRIPODD achieves state-of-the-art interpretability and competitive drift detection relative to black-box detectors, including a detailed USENET2 case study. The approach offers a practical, generic tool for monitoring and understanding distributional shifts in deployed predictive systems.

Abstract

Data in the real world often has an evolving distribution. Thus, machine learning models trained on such data get outdated over time. This phenomenon is called model drift. Knowledge of this drift serves two purposes: (i) Retain an accurate model and (ii) Discovery of knowledge or insights about change in the relationship between input features and output variable w.r.t. the model. Most existing works focus only on detecting model drift but offer no interpretability. In this work, we take a principled approach to study the problem of interpretable model drift detection from a risk perspective using a feature-interaction aware hypothesis testing framework, which enjoys guarantees on test power. The proposed framework is generic, i.e., it can be adapted to both classification and regression tasks. Experiments on several standard drift detection datasets show that our method is superior to existing interpretable methods (especially on real-world datasets) and on par with state-of-the-art black-box drift detection methods. We also quantitatively and qualitatively study the interpretability aspect including a case study on USENET2 dataset. We find our method focuses on model and drift sensitive features compared to baseline interpretable drift detectors.

Interpretable Model Drift Detection

TL;DR

Real-world data distributions evolve over time, causing model drift and potential performance degradation. The paper delivers TRIPODD, a theory-grounded, feature-interaction aware hypothesis-testing framework that uses model risk to detect and interpret drift for both classification and regression. It defines feature-sensitive model drift via and builds a hypothesis test around risk differences , with bootstrap-derived thresholds to control error. Empirical results on synthetic and real-world drift datasets show TRIPODD achieves state-of-the-art interpretability and competitive drift detection relative to black-box detectors, including a detailed USENET2 case study. The approach offers a practical, generic tool for monitoring and understanding distributional shifts in deployed predictive systems.

Abstract

Data in the real world often has an evolving distribution. Thus, machine learning models trained on such data get outdated over time. This phenomenon is called model drift. Knowledge of this drift serves two purposes: (i) Retain an accurate model and (ii) Discovery of knowledge or insights about change in the relationship between input features and output variable w.r.t. the model. Most existing works focus only on detecting model drift but offer no interpretability. In this work, we take a principled approach to study the problem of interpretable model drift detection from a risk perspective using a feature-interaction aware hypothesis testing framework, which enjoys guarantees on test power. The proposed framework is generic, i.e., it can be adapted to both classification and regression tasks. Experiments on several standard drift detection datasets show that our method is superior to existing interpretable methods (especially on real-world datasets) and on par with state-of-the-art black-box drift detection methods. We also quantitatively and qualitatively study the interpretability aspect including a case study on USENET2 dataset. We find our method focuses on model and drift sensitive features compared to baseline interpretable drift detectors.

Paper Structure

This paper contains 15 sections, 1 theorem, 4 equations, 1 figure, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background and Preliminaries
  4. Notations.
  5. TRIPODD: Methodology
  6. Hypothesis Testing Framework
  7. H$_a$:
  8. Methodology
  9. Guarantees of Test Statistic
  10. Experiments and Results
  11. Model Drift Detection
  12. Model Drift Interpretability
  13. Discussion and Analysis
  14. Conclusions
  15. Appendix

Key Result

theorem 1

The test power of the proposed hypothesis test in Definition defn:hypothesis2 converges to the ideal value 1 under the alternate hypothesis, i.e. suppose the alternate hypothesis $H_a$ is true for some feature $k\in [d]$, then, for any $t>0$, $\lim _{n\rightarrow \infty } \mathbb{P}[\widehat{c}^{k}_

Figures (1)

  • Figure 1: Qualitative Case Study for Drift Interpretability on USENET2 Dataset: Highlighted cells (green or red) are words related to drift w.r.t. each method. Green indicates a correct semantic connection to the drift, red indicates semantically irrelevant words. Our method selects relevant words with a higher sensitivity (as shown on the occlusion metric value). More details in Sec \ref{['sec:interp']}). Note:Bradley = baseball team & Ames = NASA center

Theorems & Definitions (5)

  • definition 1: Model Drift
  • definition 2: Feature-Sensitive Model Drift
  • definition 3: Hypothesis Test
  • definition 4: Test Statistic
  • theorem 1: Convergence of Test Power