The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Products

TL;DR

This paper tackles the challenge of studying ML products by creating an open-source dataset of 262 end-user OSS ML products and conducting an in-depth analysis of 30 representative products. It introduces a tailored, multi-stage search pipeline to identify ML products on GitHub and assesses collaboration, architecture, processes, testing, operations, and Responsible AI in these OSS projects. The study reports 21 findings, including limited data-scientist involvement, low modularity between ML and non-ML code, diverse architectural patterns, and scarce adoption of industry best practices like pipeline automation and monitoring, alongside seven implications for research, education, and practice. The work underscores that OSS ML products resemble startup-style development and highlights opportunities to develop tooling, educational materials, and open-source–friendly monitoring approaches to improve product quality and responsible AI practices. The dataset and insights provide a valuable resource for researchers, educators, and practitioners to analyze, reproduce, and extend ML product studies outside of restricted industry contexts.

Abstract

Machine learning (ML) components are increasingly incorporated into software products for end-users, but developers face challenges in transitioning from ML prototypes to products. Academics have limited access to the source of commercial ML products, hindering research progress to address these challenges. In this study, first and foremost, we contribute a dataset of 262 open-source ML products for end users (not just models), identified among more than half a million ML-related projects on GitHub. Then, we qualitatively and quantitatively analyze 30 open-source ML products to answer six broad research questions about development practices and system architecture. We find that the majority of the ML products in our sample represent more startup-style development than reported in past interview studies. We report 21 findings, including limited involvement of data scientists in many open-source ML products, unusually low modularity between ML and non-ML code, diverse architectural choices on incorporating models into products, and limited prevalence of industry best practices such as model testing, pipeline automation, and monitoring. Additionally, we discuss seven implications of this study on research, development, and education, including the need for tools to assist teams without data scientists, education opportunities, and open-source-specific research for privacy-preserving telemetry.

Figures (5)

• Figure :
• Figure :
• Figure :
• Figure :
• Figure :