A Benchmark Dataset And LLMs Comparison For NFR Classification With Explainable AI

TL;DR

The paper tackles automatic non-functional requirements (NFR) classification by expanding the PROMISE_exp dataset with project charters and open-source documentation, and benchmarking multiple LLMs (CodeBERT, Gemma-2, Phi-3, Mistral-8B, Llama-3.1-8B, RoBERTa) using LIME to reveal decision rationale. It demonstrates that Gemma-2 achieves the strongest performance (precision ~0.87, recall ~0.89, F1 ~0.88) with high explainability, while Phi-3 closely follows, and RoBERTa underperforms. The methodology combines an extended labeled dataset, careful prompt-based classification, and robust evaluation with traditional metrics plus LIME-based interpretability to ensure transparent NFR classification. The results underscore the value of advanced, explainable LLMs for accurate NFR categorization, with practical implications for faster, more reliable requirements engineering. The authors also propose expanding the dataset and evaluating newer LLMs to further improve robustness and explainability in real-world settings.

Abstract

Non-Functional Requirements (NFRs) play a critical role in determining the overall quality and user satisfaction of software systems. Accurately identifying and classifying NFRs is essential to ensure that software meets performance, usability, and reliability expectations. However, manual identification of NFRs from documentation is time-consuming and prone to errors, necessitating automated solutions. Before implementing any automated solution, a robust and comprehensive dataset is essential. To build such a dataset, we collected NFRs from various Project Charters and Open Source Software Documentation. This enhanced the technical depth and usability of an already existing NFR dataset. We categorized NFRs into sub-classes and identified needs using widely used Large Language Models to facilitate automation. After classifying the NFRs, we compared the classification results of the selected LLMs: RoBERTa, CodeBERT, Gemma-2, Phi-3, Mistral-8B, and Llama-3.1-8B using various evaluation metrics, including precision, recall, F1-score, and lime scores. Among these models, Gemma-2 achieved the best results with a precision of 0.87, recall of 0.89, and F1-score of 0.88, alongside a lime hit score of 78 out of 80. Phi-3 closely followed with a precision of 0.85, recall of 0.87, F1-score of 0.86, and the highest lime hit score of 79. By improving the contextual foundation, this integration enhanced the model's comprehension of technical aspects and user requirements.

Figures (4)

• Figure 1: Proposed Methodology
• Figure 2: An example of hit
• Figure 3: An example of miss
• Figure 4: Comparison of Classifiers based on Precision, Recall, and F1-score