Exploring Data Management Challenges and Solutions in Agile Software Development: A Literature Review and Practitioner Survey
Ahmed Fawzy, Amjed Tahir, Matthias Galster, Peng Liang
TL;DR
The paper systematically maps data management challenges in agile software development through a mixed-methods study (SLR of 45 studies plus a practitioner survey of 32 professionals). It identifies core challenges—data integration, capturing diverse data, automated data collection, data quality, and real-time analytics—and outlines corresponding solutions such as ontology-driven integration, data mesh governance, automation, and visualization. The findings are triangulated across academic and industry perspectives, highlighting practical implications for governance, processes, and tooling, while noting the need for empirical validation of many proposed approaches. Overall, the work provides actionable guidance for practitioners and a roadmap for researchers to enhance data management in agile environments and improve project outcomes.
Abstract
Context: Managing data related to a software product and its development poses significant challenges for software projects and agile development teams. These include integrating data from diverse sources and ensuring data quality amidst continuous change and adaptation. Objective: The paper systematically explores data management challenges and potential solutions in agile projects, aiming to provide insights into data management challenges and solutions for both researchers and practitioners. Method: We employed a mixed-methods approach, including a systematic literature review (SLR) to understand the state-of-research followed by a survey with practitioners to reflect on the state-of-practice. The SLR reviewed 45 studies, identifying and categorizing data management aspects along with their associated challenges and solutions. The practitioner survey captured practical experiences and solutions from 32 industry practitioners who were significantly involved in data management to complement the findings from the SLR. Results: Our findings identified major data management challenges in practice, such as managing data integration processes, capturing diverse data, automating data collection, and meeting real-time analysis requirements. To address the challenges, solutions such as automation tools, decentralized data management practices, and ontology-based approaches have been identified. The solutions enhance data integration, improve data quality, and enable real-time decision-making by providing flexible frameworks tailored to agile project needs. Conclusion: The study pinpointed significant challenges and actionable solutions in data management for agile software development. Our findings provide practical implications for practitioners and researchers, emphasizing the development of effective data management practices and tools to address those challenges and improve project success.