From Online User Feedback to Requirements: Evaluating Large Language Models for Classification and Specification Tasks

TL;DR

Online user feedback is a rich but noisy source for requirements engineering, and prior work lacks systematic evaluation and replication. The authors empirically evaluate five lightweight open‑source LLMs (Llama 2, Llama 3, Mistral, Gemma 2, Phi‑3 Mini) on three RE tasks—classifying feedback by user request type ($t_{UR}$) and by NFR type ($t_{NFR}$), and generating formal requirement specifications—using five prompting strategies. Classification performance centers around $F1$ in the range $0.40$–$0.74$ (UR up to $0.74$, NFR up to $0.55$), with few‑shot prompting often yielding the best UR results and chain‑of‑thought helping NFR classification; specification generation achieves mean quality near $3.1$ out of $5$, with some models reaching $3.6$. The work contributes a replication package and insights into the capabilities and limits of lightweight LLMs for RE, suggesting they can assist early RE tasks but require human oversight and context adaptation for reliable industrial deployment.

Abstract

[Context and Motivation] Online user feedback provides valuable information to support requirements engineering (RE). However, analyzing online user feedback is challenging due to its large volume and noise. Large language models (LLMs) show strong potential to automate this process and outperform previous techniques. They can also enable new tasks, such as generating requirements specifications. [Question-Problem] Despite their potential, the use of LLMs to analyze user feedback for RE remains underexplored. Existing studies offer limited empirical evidence, lack thorough evaluation, and rarely provide replication packages, undermining validity and reproducibility. [Principal Idea-Results] We evaluate five lightweight open-source LLMs on three RE tasks: user request classification, NFR classification, and requirements specification generation. Classification performance was measured on two feedback datasets, and specification quality via human evaluation. LLMs achieved moderate-to-high classification accuracy (F1 ~ 0.47-0.68) and moderately high specification quality (mean ~ 3/5). [Contributions] We newly explore lightweight LLMs for feedback-driven requirements development. Our contributions are: (i) an empirical evaluation of lightweight LLMs on three RE tasks, (ii) a replication package, and (iii) insights into their capabilities and limitations for RE.