Modyn: Data-Centric Machine Learning Pipeline Orchestration

TL;DR

This paper tackles the challenge of retraining models on continuously growing datasets with potential distribution shifts by proposing Modyn, a data-centric pipeline orchestrator. Modyn provides declarative data-selection and triggering policies, enabling continuous retraining while minimizing data processed and the number of retraining events. The authors formalize pipeline evaluation through composite models and an evaluation matrix, and demonstrate through a benchmark suite and experiments that sample-level data selection and drift-based triggering can reduce training cost without sacrificing accuracy, while achieving high throughput via careful data retrieval optimizations. The work presents a practical, open-source platform with an ecosystem of datasets, models, and tooling that supports policy exploration and end-to-end pipeline management, with implications for MLOps in dynamic data environments.

Abstract

In real-world machine learning (ML) pipelines, datasets are continuously growing. Models must incorporate this new training data to improve generalization and adapt to potential distribution shifts. The cost of model retraining is proportional to how frequently the model is retrained and how much data it is trained on, which makes the naive approach of retraining from scratch each time impractical. We present Modyn, a data-centric end-to-end machine learning platform. Modyn's ML pipeline abstraction enables users to declaratively describe policies for continuously training a model on a growing dataset. Modyn pipelines allow users to apply data selection policies (to reduce the number of data points) and triggering policies (to reduce the number of trainings). Modyn executes and orchestrates these continuous ML training pipelines. The system is open-source and comes with an ecosystem of benchmark datasets, models, and tooling. We formally discuss how to measure the performance of ML pipelines by introducing the concept of composite models, enabling fair comparison of pipelines with different data selection and triggering policies. We empirically analyze how various data selection and triggering policies impact model accuracy, and also show that Modyn enables high throughput training with sample-level data selection.

Figures (16)

• Figure 1: Mean accuracies of 9 selection strategies (50 % subset) and full data training (see \ref{['subsubsec:eval-selection-yearbook']}).
• Figure 2: Visualization of finding the currently active model.
• Figure 3: Modyn's system design.
• Figure 4: Excerpt from an example Modyn pipeline.
• Figure 5: Data selection flow in Modyn.