Applying a Requirements-Focused Agile Management Approach for Machine Learning-Enabled Systems

TL;DR

The paper tackles the challenge of aligning requirements engineering and agile management with ML-enabled systems, where data dependence and model experimentation complicate traditional practices. It introduces RefineML, a synthesis of PerSpecML, Agile4MLS, and Lean R&D, augmented by Minimal Valuable Model (MVM), Levels of Done (LoD), and a Demo API to steer continuous refinement and governance. The approach is evaluated in a real industry–academia project (EXA and PUC-Rio), using TAM-based surveys and thematic analysis, revealing high perceived usefulness and acceptance, improved communication, and effective dual-track governance, while highlighting persistent difficulties in translating PerSpecML into ML backlog and in estimating ML effort. The work provides a practical blueprint for managing ML-enabled development with continuous requirements refinement and points to platform engineering as a key area for further improvement to support ML operations.

Abstract

Machine Learning (ML)-enabled systems challenge traditional Requirements Engineering (RE) and agile management due to data dependence, experimentation, and uncertain model behavior. Existing RE and agile practices remain poorly integrated and insufficiently tailored to these characteristics. This paper reports on the practical experience of applying RefineML, a requirements-focused approach for the continuous and agile refinement of ML-enabled systems, which integrates ML-tailored specification and agile management approaches with best practices derived from a systematic mapping study. The application context concerns an industry-academia collaboration project between PUC-Rio and EXA, a Brazilian cybersecurity company. For evaluation purposes, we applied questionnaires assessing RefineML's suitability and overall acceptance and semi-structured interviews. We applied thematic analysis to the collected qualitative data. Regarding suitability and acceptance, the results of the questionnaires indicated high perceived usefulness and intention to use. Based on the interviews, stakeholders perceived RefineML as improving communication and facilitating early feasibility assessments, as well as enabling dual-track governance of ML and software work, allowing continuous refinement of the model while evolving the overall software project. However, some limitations remain, particularly related to difficulties in operationalizing ML concerns into agile requirements and in estimating ML effort.

Figures (5)

• Figure 1: PerSpecML concerns illustration diagram villamizar2024identifying.
• Figure 2: The sprint planning process of Agile4MLS vaidhyanathan2022agile4mls.
• Figure 3: RefineML approach.
• Figure 4: Suitability of RefineML phases according to the Development Team.
• Figure 5: Technology acceptance responses from the development team.