From Bugs to Benefits: Improving User Stories by Leveraging Crowd Knowledge with CrUISE-AC
Stefan Schwedt, Thomas Ströder
TL;DR
CrUISE-AC presents a fully automated pipeline that leverages crowd knowledge from issue trackers to enrich user stories with non-obvious acceptance criteria. It combines preprocessing, zero-shot matching with an ensemble of decoder-only LLMs, and generation of Gherkin-form acceptance criteria via GPT-4 Turbo, followed by novelty checks and expert evaluation. Across ecommerce and CMS domains, the approach yields AC additions that experts deem useful in the majority of cases (≈82% in ecommerce, ≈80% in CMS), demonstrating the value of external crowd data for requirements engineering. Limitations include dependence on available input issues and the absence of cross-story conflict checks, with future work targeting scalability, weighting of models, and field studies to validate industrial applicability.
Abstract
Costs for resolving software defects increase exponentially in late stages. Incomplete or ambiguous requirements are one of the biggest sources for defects, since stakeholders might not be able to communicate their needs or fail to share their domain specific knowledge. Combined with insufficient developer experience, teams are prone to constructing incorrect or incomplete features. To prevent this, requirements engineering has to explore knowledge sources beyond stakeholder interviews. Publicly accessible issue trackers for systems within the same application domain hold essential information on identified weaknesses, edge cases, and potential error sources, all documented by actual users. Our research aims at (1) identifying, and (2) leveraging such issues to improve an agile requirements artifact known as a "user story". We present CrUISE-AC (Crowd and User Informed Suggestion Engine for Acceptance Criteria) as a fully automated method that investigates issues and generates non-trivial additional acceptance criteria for a given user story by employing NLP techniques and an ensemble of LLMs. CrUISE- AC was evaluated by five independent experts in two distinct business domains. Our findings suggest that issue trackers hold valuable information pertinent to requirements engineering. Our evaluation shows that 80-82% of the generated acceptance criteria add relevant requirements to the user stories. Limitations are the dependence on accessible input issues and the fact that we do not check generated criteria for being conflict-free or non-overlapping with criteria from other user stories.