Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization

Fiorella Zampetti, Federico Stocchetti, Federica Razzano, Damian Andrew Tamburri, Massimiliano Di Penta

TL;DR

This paper addresses how open-source MLOps frameworks are used in practice by analyzing 969 Python GitHub projects importing eight popular frameworks. The authors extract usage patterns from code and GitHub workflows, and build a taxonomy of 14 feature-request categories (37 sub-categories) using manual annotation and LLM-assisted classification of 4,386 issues. The results show that frameworks are primarily used as APIs, with limited out-of-the-box or CI/CD usage, and that users often combine multiple frameworks to cover different lifecycle phases. The study provides actionable insights for framework developers and practitioners, and offers a replication package to support future research.

Abstract

Machine Learning (ML) Operations (MLOps) frameworks have been conceived to support developers and AI engineers in managing the lifecycle of their ML models. While such frameworks provide a wide range of features, developers may leverage only a subset of them, while missing some highly desired features. This paper investigates the practical use and desired feature enhancements of eight popular open-source MLOps frameworks. Specifically, we analyze their usage by dependent projects on GitHub, examining how they invoke the frameworks' APIs and commands. Then, we qualitatively analyze feature requests and enhancements mined from the frameworks' issue trackers, relating these desired improvements to the previously identified usage features. Results indicate that MLOps frameworks are rarely used out-of-the-box and are infrequently integrated into GitHub Workflows, but rather, developers use their APIs to implement custom functionality in their projects. Used features concern core ML phases and whole infrastructure governance, sometimes leveraging multiple frameworks with complementary features. The mapping with feature requests highlights that users mainly ask for enhancements to core features of the frameworks, but also better API exposure and CI/CD integration.

How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization

TL;DR

This paper addresses how open-source MLOps frameworks are used in practice by analyzing 969 Python GitHub projects importing eight popular frameworks. The authors extract usage patterns from code and GitHub workflows, and build a taxonomy of 14 feature-request categories (37 sub-categories) using manual annotation and LLM-assisted classification of 4,386 issues. The results show that frameworks are primarily used as APIs, with limited out-of-the-box or CI/CD usage, and that users often combine multiple frameworks to cover different lifecycle phases. The study provides actionable insights for framework developers and practitioners, and offers a replication package to support future research.

Abstract

Machine Learning (ML) Operations (MLOps) frameworks have been conceived to support developers and AI engineers in managing the lifecycle of their ML models. While such frameworks provide a wide range of features, developers may leverage only a subset of them, while missing some highly desired features. This paper investigates the practical use and desired feature enhancements of eight popular open-source MLOps frameworks. Specifically, we analyze their usage by dependent projects on GitHub, examining how they invoke the frameworks' APIs and commands. Then, we qualitatively analyze feature requests and enhancements mined from the frameworks' issue trackers, relating these desired improvements to the previously identified usage features. Results indicate that MLOps frameworks are rarely used out-of-the-box and are infrequently integrated into GitHub Workflows, but rather, developers use their APIs to implement custom functionality in their projects. Used features concern core ML phases and whole infrastructure governance, sometimes leveraging multiple frameworks with complementary features. The mapping with feature requests highlights that users mainly ask for enhancements to core features of the frameworks, but also better API exposure and CI/CD integration.
Paper Structure (19 sections, 2 figures, 5 tables)

This paper contains 19 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. MLOps Frameworks: Background
  3. Empirical Study
  4. MLOps framework Selection
  5. Dataset Collection
  6. Usage Extraction
  7. Usage Classification
  8. Feature Request Analysis
  9. Study Results
  10. RQ$_1$: MLOps Frameworks Usage
  11. RQ$_2$: Frameworks Features Being Used
  12. RQ$_3$: MLOps Frameworks Feature Requests
  13. How feature usages relate to feature requests
  14. Threats to Validity
  15. Related Work
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Data analysis process.
  • Figure 2: Taxonomy of feature requests from MLOps frameworks issue trackers.