Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TraceLLM: Leveraging Large Language Models with Prompt Engineering for Enhanced Requirements Traceability

Nouf Alturayeif, Irfan Ahmad, Jameleddine Hassine

TL;DR

TraceLLM addresses the challenge of automated requirements traceability by systematically engineering prompts and demonstration strategies for large language models. The authors propose a four-stage framework—dataset splitting, core prompt design, prompt enrichment, and evaluation—evaluated across eight state-of-the-art LLMs and four diverse datasets, achieving state-of-the-art F2 scores and highlighting the impact of demonstration selection. They demonstrate cross-model generalization, analyze various DSS approaches, and show that prompt quality often outweighs model size, with lightweight models performing competitively when paired with well-crafted prompts. The work advocates semi-automated traceability workflows that assist human analysts and provides replication materials, underscoring practical viability and cost-aware deployment paths.

Abstract

Requirements traceability, the process of establishing and maintaining relationships between requirements and various software development artifacts, is paramount for ensuring system integrity and fulfilling requirements throughout the Software Development Life Cycle (SDLC). Traditional methods, including manual and information retrieval models, are labor-intensive, error-prone, and limited by low precision. Recently, Large Language Models (LLMs) have demonstrated potential for supporting software engineering tasks through advanced language comprehension. However, a substantial gap exists in the systematic design and evaluation of prompts tailored to extract accurate trace links. This paper introduces TraceLLM, a systematic framework for enhancing requirements traceability through prompt engineering and demonstration selection. Our approach incorporates rigorous dataset splitting, iterative prompt refinement, enrichment with contextual roles and domain knowledge, and evaluation across zero- and few-shot settings. We assess prompt generalization and robustness using eight state-of-the-art LLMs on four benchmark datasets representing diverse domains (aerospace, healthcare) and artifact types (requirements, design elements, test cases, regulations). TraceLLM achieves state-of-the-art F2 scores, outperforming traditional IR baselines, fine-tuned models, and prior LLM-based methods. We also explore the impact of demonstration selection strategies, identifying label-aware, diversity-based sampling as particularly effective. Overall, our findings highlight that traceability performance depends not only on model capacity but also critically on the quality of prompt engineering. In addition, the achieved performance suggests that TraceLLM can support semi-automated traceability workflows in which candidate links are reviewed and validated by human analysts.

TraceLLM: Leveraging Large Language Models with Prompt Engineering for Enhanced Requirements Traceability

TL;DR

TraceLLM addresses the challenge of automated requirements traceability by systematically engineering prompts and demonstration strategies for large language models. The authors propose a four-stage framework—dataset splitting, core prompt design, prompt enrichment, and evaluation—evaluated across eight state-of-the-art LLMs and four diverse datasets, achieving state-of-the-art F2 scores and highlighting the impact of demonstration selection. They demonstrate cross-model generalization, analyze various DSS approaches, and show that prompt quality often outweighs model size, with lightweight models performing competitively when paired with well-crafted prompts. The work advocates semi-automated traceability workflows that assist human analysts and provides replication materials, underscoring practical viability and cost-aware deployment paths.

Abstract

Requirements traceability, the process of establishing and maintaining relationships between requirements and various software development artifacts, is paramount for ensuring system integrity and fulfilling requirements throughout the Software Development Life Cycle (SDLC). Traditional methods, including manual and information retrieval models, are labor-intensive, error-prone, and limited by low precision. Recently, Large Language Models (LLMs) have demonstrated potential for supporting software engineering tasks through advanced language comprehension. However, a substantial gap exists in the systematic design and evaluation of prompts tailored to extract accurate trace links. This paper introduces TraceLLM, a systematic framework for enhancing requirements traceability through prompt engineering and demonstration selection. Our approach incorporates rigorous dataset splitting, iterative prompt refinement, enrichment with contextual roles and domain knowledge, and evaluation across zero- and few-shot settings. We assess prompt generalization and robustness using eight state-of-the-art LLMs on four benchmark datasets representing diverse domains (aerospace, healthcare) and artifact types (requirements, design elements, test cases, regulations). TraceLLM achieves state-of-the-art F2 scores, outperforming traditional IR baselines, fine-tuned models, and prior LLM-based methods. We also explore the impact of demonstration selection strategies, identifying label-aware, diversity-based sampling as particularly effective. Overall, our findings highlight that traceability performance depends not only on model capacity but also critically on the quality of prompt engineering. In addition, the achieved performance suggests that TraceLLM can support semi-automated traceability workflows in which candidate links are reviewed and validated by human analysts.
Paper Structure (50 sections, 8 equations, 12 figures, 10 tables, 4 algorithms)

This paper contains 50 sections, 8 equations, 12 figures, 10 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-Based Traceability Methodology
  4. Dataset Splitting
  5. Core Prompt Design
  6. Prompt Enrichment
  7. Evaluation
  8. Demonstrations Selection Strategies
  9. Diversity-Based Strategy
  10. Formulation
  11. Implementation Algorithm
  12. Similarity-Based Selection
  13. Formulation
  14. Implementation Algorithm
  15. Uncertainty-Based Selection
  16. ...and 35 more sections

Figures (12)

  • Figure 1: LLM-based traceability methodology (TraceLLM)
  • Figure 2: T-SNE visualization of the demonstrations selected by the different selection strategies. The rows indicate balanced and unbalanced selection, while the columns represent different strategies.
  • Figure 3: Initial prompt and the LLM response.
  • Figure 4: Prompt with context information.
  • Figure 5: Prompt with artifacts-related information.
  • ...and 7 more figures