Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Expert-Driven Monitoring of Operational ML Models

Joran Leest, Claudia Raibulet, Ilias Gerostathopoulos, Patricia Lago

TL;DR

This work addresses the challenge of maintaining reliable ML performance under concept drift by advocating Expert Monitoring, which embeds domain expertise into drift detection and response. It proposes a scenario-based framework where experts elicit drift-inducing events, formalize them using a standardized format, and apply Bayesian model comparison to identify which scenario explains observed drift, potentially triggering automated mitigations. The key contributions include a structured elicitation workflow, a Bayesian interpretation of drift scenarios with $P(M|D)$ and Bayes factors, and a pathway toward centralized, on-call governance of drift responses. The approach aims to reduce alert fatigue, improve interpretability of drift causes, and strengthen MLOps practices by combining human expertise with probabilistic reasoning for operational ML systems.

Abstract

We propose Expert Monitoring, an approach that leverages domain expertise to enhance the detection and mitigation of concept drift in machine learning (ML) models. Our approach supports practitioners by consolidating domain expertise related to concept drift-inducing events, making this expertise accessible to on-call personnel, and enabling automatic adaptability with expert oversight.

Expert-Driven Monitoring of Operational ML Models

TL;DR

This work addresses the challenge of maintaining reliable ML performance under concept drift by advocating Expert Monitoring, which embeds domain expertise into drift detection and response. It proposes a scenario-based framework where experts elicit drift-inducing events, formalize them using a standardized format, and apply Bayesian model comparison to identify which scenario explains observed drift, potentially triggering automated mitigations. The key contributions include a structured elicitation workflow, a Bayesian interpretation of drift scenarios with and Bayes factors, and a pathway toward centralized, on-call governance of drift responses. The approach aims to reduce alert fatigue, improve interpretability of drift causes, and strengthen MLOps practices by combining human expertise with probabilistic reasoning for operational ML systems.

Abstract

We propose Expert Monitoring, an approach that leverages domain expertise to enhance the detection and mitigation of concept drift in machine learning (ML) models. Our approach supports practitioners by consolidating domain expertise related to concept drift-inducing events, making this expertise accessible to on-call personnel, and enabling automatic adaptability with expert oversight.
Paper Structure (18 sections, 2 equations, 4 figures)

This paper contains 18 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Navigating the Darkness: Concept Drift in Practice
  3. The Practical Challenges of Concept Drift Detection -- Where the Shadows Lie
  4. Concept Drift Detection Without Labeled Data
  5. Deciphering the Nature of Concept drift Post-Detection
  6. Domain Expertise -- A Light in Dark Places, When All Other Lights Go Out
  7. Addressing Concept Drift with Domain Expertise
  8. Why Is It Hard to Rely on Domain Expertise?
  9. Approach
  10. Scenario Specification
  11. Expert Knowledge Elicitation
  12. Retrospective Analysis
  13. Scenario Specification Format
  14. Scenario Identification
  15. Bayesian Model Comparison
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Data drift in a customer churn prediction model.
  • Figure 2: An illustrative case for customer churn prediction, showing expert assessments for three cases of feature drift.
  • Figure 3: A visual depiction of our approach, Expert Monitoring. In step (A), ML engineers consolidate domain expertise within the organization using a standardized format. In step (B), upon detecting feature drift, scenarios are evaluated using Bayesian model comparison. Afterwards, the ML engineer is informed about potential causes, or an automated response is triggered.
  • Figure 4: Detection accuracy vs. estimate uncertainty and error (in proportion relative to actual parameter values) on simulated scenarios, with a Bayes factor threshold of 5.