USeR: A Web-based User Story eReviewer for Assisted Quality Optimizations

TL;DR

USeR tackles the problem of inconsistent and hard-to-interpret user story quality in agile development by delivering a web-based reviewer that computes 34 metrics from an initial 77 through expert refinement, and implements eight core metrics with ML-based and rule-based methods. It provides a RESTful API and a user interface that delivers instant, explainable feedback to guide quality optimization. An empirical expert study on 100 user stories from automotive and health projects demonstrates that the tool can capture substantial portions of expert judgment (R^2 up to 0.76 in automotive) and identifies Format Complete and Word Sparse as the strongest predictors of perceived quality. The work establishes a foundation for integrated, explainable assistants in requirements engineering and outlines concrete future directions, including inline issue highlighting and LLM-assisted recommendations, for closer integration into developers’ daily tooling.

Abstract

User stories are widely applied for conveying requirements within agile software development teams. Multiple user story quality guidelines exist, but authors like Product Owners in industry projects frequently fail to write high-quality user stories. This situation is exacerbated by the lack of tools for assessing user story quality. In this paper, we propose User Story eReviewer (USeR) a web-based tool that allows authors to determine and optimize user story quality. For developing USeR, we collected 77 potential quality metrics through literature review, practitioner sessions, and research group meetings and refined these to 34 applicable metrics through expert sessions. Finally, we derived algorithms for eight prioritized metrics using a literature review and research group meetings and implemented them with plain code and machine learning techniques. USeR offers a RESTful API and user interface for instant, consistent, and explainable user feedback supporting fast and easy quality optimizations. It has been empirically evaluated with an expert study using 100 user stories and four experts from two real-world agile software projects in the automotive and health sectors.

Figures (6)

• Figure 1: Conceptual framework of USeR. The system facilitates an iterative process where authors receive automated quality feedback on user stories and subsequently implement optimizations to improve their quality.
• Figure 2: Architecture of USeR containing the web-based user interface and the API as backend. Virtualization is used to host the user interface and the API. USeR implemented a Representational State Transfer (RESTful) API to facilitate standardized communication with the web app.
• Figure 3: USeR's API parts: data importer that covers text cleaning and converting, training models and metrics preparation, prediction of the metrics values, and interpretation with calculating the percentiles as orientation for authors to plan their user story quality optimizations.
• Figure 4: Web-based user interface of USeR, which is structured into the user story input area and the quality metrics part. Authors can edit user stories in the text box and the quality metrics show the quality ratings of a user story.
• Figure 5: Automotive experts' statistics show an increased medium-quality rating tendency and a strong inter-rater reliability.