A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development
Werner Geyer, Jessica He, Daita Sarkar, Michelle Brachman, Chris Hammond, Jennifer Heins, Zahra Ashktorab, Carlos Rosemberg, Charlie Hill
TL;DR
The study addresses the problem of poorly defined agile epics and investigates whether large language models can support epic quality evaluation. It reports the development of a rubric with eight elements and an LLM-based Epic Evaluator, validated through a qualitative study with 17 product managers in a global company. Findings show that LLM-based epic evaluations are viable and valued when domain knowledge, stage-awareness, and tool integration are addressed, though customization and human oversight remain essential. The work demonstrates practical design principles for integrating AI-assisted epic evaluation into agile workflows, with implications for practitioners and researchers seeking to improve epic quality and collaboration.
Abstract
The broad availability of generative AI offers new opportunities to support various work domains, including agile software development. Agile epics are a key artifact for product managers to communicate requirements to stakeholders. However, in practice, they are often poorly defined, leading to churn, delivery delays, and cost overruns. In this industry case study, we investigate opportunities for large language models (LLMs) to evaluate agile epic quality in a global company. Results from a user study with 17 product managers indicate how LLM evaluations could be integrated into their work practices, including perceived values and usage in improving their epics. High levels of satisfaction indicate that agile epics are a new, viable application of AI evaluations. However, our findings also outline challenges, limitations, and adoption barriers that can inform both practitioners and researchers on the integration of such evaluations into future agile work practices.