How Effective are Generative Large Language Models in Performing Requirements Classification?
Waad Alhoshan, Alessio Ferrari, Liping Zhao
TL;DR
This paper systematically evaluates generative large language models (Bloom, Gemma, Llama) for requirements classification across three RE datasets, using both binary and multi-class tasks and a comprehensive suite of prompts and dataset variations. By applying inference-based learning for generative models and embedding-based learning for non-generative baselines, the study reveals that prompt design and model architecture universally influence performance, while dataset variations have more nuanced effects tied to task complexity. Across over 400 experiments, findings show no single generative model dominates all tasks; in binary classifications, performance is task-dependent, whereas in multi-class classifications, non-generative baselines (e.g., All-Mini) often outperform generative LLMs. The work provides actionable guidance on selecting models and crafting prompts for RE classification and establishes a baseline for future cross-domain and model-scale investigations.
Abstract
In recent years, transformer-based large language models (LLMs) have revolutionised natural language processing (NLP), with generative models opening new possibilities for tasks that require context-aware text generation. Requirements engineering (RE) has also seen a surge in the experimentation of LLMs for different tasks, including trace-link detection, regulatory compliance, and others. Requirements classification is a common task in RE. While non-generative LLMs like BERT have been successfully applied to this task, there has been limited exploration of generative LLMs. This gap raises an important question: how well can generative LLMs, which produce context-aware outputs, perform in requirements classification? In this study, we explore the effectiveness of three generative LLMs-Bloom, Gemma, and Llama-in performing both binary and multi-class requirements classification. We design an extensive experimental study involving over 400 experiments across three widely used datasets (PROMISE NFR, Functional-Quality, and SecReq). Our study concludes that while factors like prompt design and LLM architecture are universally important, others-such as dataset variations-have a more situational impact, depending on the complexity of the classification task. This insight can guide future model development and deployment strategies, focusing on optimising prompt structures and aligning model architectures with task-specific needs for improved performance.