Continuous Integration Practices in Machine Learning Projects: The Practitioners` Perspective
João Helis Bernardo, Daniel Alencar da Costa, Filipe Roseiro Cogo, Sérgio Queiróz de Medeiros, Uirá Kulesza
TL;DR
The paper addresses how CI practices must adapt to ML projects by exploring practitioner perspectives that reveal longer build times and lower test coverage compared with traditional software. It combines prior quantitative insights with a qualitative survey of 155 practitioners across 47 ML projects, uncovering eight interrelated barriers such as testing complexity, data handling, and infrastructure needs. It proposes ML-specific CI practices—track model performance per commit, granular testing, dataset/model versioning, handling non-determinism, and caching—to bridge the gap between conventional CI and ML realities, and emphasizes interdisciplinary collaboration to strengthen testing culture. The findings offer actionable guidance for more efficient, robust CI in ML and lay groundwork for standardized, domain-tailored CI guidelines in MLOps practice.
Abstract
Continuous Integration (CI) is a cornerstone of modern software development. However, while widely adopted in traditional software projects, applying CI practices to Machine Learning (ML) projects presents distinctive characteristics. For example, our previous work revealed that ML projects often experience longer build durations and lower test coverage rates compared to their non-ML counterparts. Building on these quantitative findings, this study surveys 155 practitioners from 47 ML projects to investigate the underlying reasons for these distinctive characteristics through a qualitative perspective. Practitioners highlighted eight key differences, including test complexity, infrastructure requirements, and build duration and stability. Common challenges mentioned by practitioners include higher project complexity, model training demands, extensive data handling, increased computational resource needs, and dependency management, all contributing to extended build durations. Furthermore, ML systems' non-deterministic nature, data dependencies, and computational constraints were identified as significant barriers to effective testing. The key takeaway from this study is that while foundational CI principles remain valuable, ML projects require tailored approaches to address their unique challenges. To bridge this gap, we propose a set of ML-specific CI practices, including tracking model performance metrics and prioritizing test execution within CI pipelines. Additionally, our findings highlight the importance of fostering interdisciplinary collaboration to strengthen the testing culture in ML projects. By bridging quantitative findings with practitioners' insights, this study provides a deeper understanding of the interplay between CI practices and the unique demands of ML projects, laying the groundwork for more efficient and robust CI strategies in this domain.