From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering

TL;DR

This study investigates using LLMs to support qualitative data analysis in Requirements Engineering by comparing inductive (zero-shot) and deductive (one/few-shot) annotation across two case studies. GPT-4 consistently outperforms other models in deductive settings, achieving substantial agreement with human analysts (up to Cohen's Kappa around $0.738$) while zero-shot performance remains limited, underscoring the value of task guidance. Detailed, context-rich prompts significantly enhance annotation accuracy and consistency, and the resulting structured labels enable traceability and direct use in domain models, reducing manual effort. While promising for scalable QDA in RE, the approach shows limitations in inductive tasks and requires broader validation across diverse RE contexts and careful bias mitigation.

Abstract

Requirements Engineering (RE) is essential for developing complex and regulated software projects. Given the challenges in transforming stakeholder inputs into consistent software designs, Qualitative Data Analysis (QDA) provides a systematic approach to handling free-form data. However, traditional QDA methods are time-consuming and heavily reliant on manual effort. In this paper, we explore the use of Large Language Models (LLMs), including GPT-4, Mistral, and LLaMA-2, to improve QDA tasks in RE. Our study evaluates LLMs' performance in inductive (zero-shot) and deductive (one-shot, few-shot) annotation tasks, revealing that GPT-4 achieves substantial agreement with human analysts in deductive settings, with Cohen's Kappa scores exceeding 0.7, while zero-shot performance remains limited. Detailed, context-rich prompts significantly improve annotation accuracy and consistency, particularly in deductive scenarios, and GPT-4 demonstrates high reliability across repeated runs. These findings highlight the potential of LLMs to support QDA in RE by reducing manual effort while maintaining annotation quality. The structured labels automatically provide traceability of requirements and can be directly utilized as classes in domain models, facilitating systematic software design.

Figures (1)

• Figure 1: Overview of the methodology integrating LLMs into QDA for RE. The process includes collecting requirement statements, designing prompts, feeding them to LLMs, and generating output labels.