Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-based agents for automating the enhancement of user story quality: An early report

Zheying Zhang, Maruf Rayhan, Tomas Herda, Manuel Goisauf, Pekka Abrahamsson

TL;DR

The paper investigates automating user story quality enhancement in agile software development using Autonomous LLM-based Agent System (ALAS). It proposes a reference model for LLM-based agents and implements it within Austrian Post Group IT, evaluating improvements to 25 synthetic mobile-delivery user stories across six teams. Results show that ALAS can improve clarity, completeness, and perceived business value of user stories, but highlight the necessity of human oversight and careful prompt design to avoid scope creep and hallucinations. The work provides a concrete proof-of-concept for AI-assisted requirements engineering in industry and outlines a roadmap for more specialized agents and refined prompts to bridge AI capabilities with agile practice.

Abstract

In agile software development, maintaining high-quality user stories is crucial, but also challenging. This study explores the use of large language models to automatically improve the user story quality in Austrian Post Group IT agile teams. We developed a reference model for an Autonomous LLM-based Agent System and implemented it at the company. The quality of user stories in the study and the effectiveness of these agents for user story quality improvement was assessed by 11 participants across six agile teams. Our findings demonstrate the potential of LLMs in improving user story quality, contributing to the research on AI role in agile development, and providing a practical example of the transformative impact of AI in an industry setting.

LLM-based agents for automating the enhancement of user story quality: An early report

TL;DR

The paper investigates automating user story quality enhancement in agile software development using Autonomous LLM-based Agent System (ALAS). It proposes a reference model for LLM-based agents and implements it within Austrian Post Group IT, evaluating improvements to 25 synthetic mobile-delivery user stories across six teams. Results show that ALAS can improve clarity, completeness, and perceived business value of user stories, but highlight the necessity of human oversight and careful prompt design to avoid scope creep and hallucinations. The work provides a concrete proof-of-concept for AI-assisted requirements engineering in industry and outlines a roadmap for more specialized agents and refined prompts to bridge AI capabilities with agile practice.

Abstract

In agile software development, maintaining high-quality user stories is crucial, but also challenging. This study explores the use of large language models to automatically improve the user story quality in Austrian Post Group IT agile teams. We developed a reference model for an Autonomous LLM-based Agent System and implemented it at the company. The quality of user stories in the study and the effectiveness of these agents for user story quality improvement was assessed by 11 participants across six agile teams. Our findings demonstrate the potential of LLMs in improving user story quality, contributing to the research on AI role in agile development, and providing a practical example of the transformative impact of AI in an industry setting.
Paper Structure (17 sections, 5 figures, 2 tables)

This paper contains 17 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. User Story Quality
  3. Methodology
  4. A Reference Model of LLM-based Agents
  5. Implementing an Agent Environment
  6. Task Preparation Phase
  7. Task Conduction Phase
  8. Experiments
  9. Setting up Experiments
  10. Task and Context of Task
  11. Agent profiles
  12. Subtasks
  13. GPT models
  14. Evaluation
  15. Results
  16. ...and 2 more sections

Figures (5)

  • Figure 1: A reference model of an agent system
  • Figure 2: User story 1 (US1) - a user story example in the Mobile Delivery project
  • Figure 3: An example excerpted from the generated AI plan
  • Figure 4: AI plan illustrated in the task conduction phase
  • Figure 5: Distribution of survey participants' perceptions of user story quality, Note: US* = User Story *, v.1 = Version 1 Improved by gpt-3.5-Turbo, v.2 = Version 2 Improved by gpt-4-1106-Preview