Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Continuous Experiment-driven MLOps

Keerthiga Rajenthiram, Milad Abdullah, Ilias Gerostathopoulos, Petr Hnetynka, Tomáš Bureš, Gerard Pons, Besim Bilalli, Anna Queralt

TL;DR

The paper addresses the brittleness and inefficiency of current ML development workflows by proposing an experiment-driven MLOps paradigm that emphasizes full traceability, repeatability, and continuous improvement through structured experiments. It introduces the ExtremeXP framework, built around Experiments and Complex Analytics Workflows (CAWs), a Knowledge Repository implemented as a Knowledge Graph, and KG-based learning with embeddings and link prediction to learn from past efforts. Human-in-the-loop control is preserved via supervisor/validator roles and interaction budgets, enabling targeted intervention while automating routine optimization. The framework demonstrates real-world applicability across multiple domains and outlines a path toward continuous, knowledge-driven evolution of ML-enabled systems in production.

Abstract

Despite advancements in MLOps and AutoML, ML development still remains challenging for data scientists. First, there is poor support for and limited control over optimizing and evolving ML models. Second, there is lack of efficient mechanisms for continuous evolution of ML models which would leverage the knowledge gained in previous optimizations of the same or different models. We propose an experiment-driven MLOps approach which tackles these problems. Our approach relies on the concept of an experiment, which embodies a fully controllable optimization process. It introduces full traceability and repeatability to the optimization process, allows humans to be in full control of it, and enables continuous improvement of the ML system. Importantly, it also establishes knowledge, which is carried over and built across a series of experiments and allows for improving the efficiency of experimentation over time. We demonstrate our approach through its realization and application in the ExtremeXP1 project (Horizon Europe).

Towards Continuous Experiment-driven MLOps

TL;DR

The paper addresses the brittleness and inefficiency of current ML development workflows by proposing an experiment-driven MLOps paradigm that emphasizes full traceability, repeatability, and continuous improvement through structured experiments. It introduces the ExtremeXP framework, built around Experiments and Complex Analytics Workflows (CAWs), a Knowledge Repository implemented as a Knowledge Graph, and KG-based learning with embeddings and link prediction to learn from past efforts. Human-in-the-loop control is preserved via supervisor/validator roles and interaction budgets, enabling targeted intervention while automating routine optimization. The framework demonstrates real-world applicability across multiple domains and outlines a path toward continuous, knowledge-driven evolution of ML-enabled systems in production.

Abstract

Despite advancements in MLOps and AutoML, ML development still remains challenging for data scientists. First, there is poor support for and limited control over optimizing and evolving ML models. Second, there is lack of efficient mechanisms for continuous evolution of ML models which would leverage the knowledge gained in previous optimizations of the same or different models. We propose an experiment-driven MLOps approach which tackles these problems. Our approach relies on the concept of an experiment, which embodies a fully controllable optimization process. It introduces full traceability and repeatability to the optimization process, allows humans to be in full control of it, and enables continuous improvement of the ML system. Importantly, it also establishes knowledge, which is carried over and built across a series of experiments and allows for improving the efficiency of experimentation over time. We demonstrate our approach through its realization and application in the ExtremeXP1 project (Horizon Europe).

Paper Structure

This paper contains 12 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. MLOps and AutoML
  4. Continuous experimentation
  5. Related work
  6. Motivating Example
  7. ExtremeXP Framework
  8. Main concepts and architecture
  9. Learning from the past
  10. Keeping human in control
  11. Continuously improving
  12. Applications and Experience

Figures (2)

  • Figure 1: Experiment Workflow Designer Tool of ExtremeXP portal.
  • Figure 2: Main concepts of ExtremeXP framework.