Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models

Abdelkarim El-Hajjami, Camille Salinesi

TL;DR

Synthline introduces a minimal Product Line approach to generate synthetic RE data using large language models, controlled by a Feature Model to support classification-based use cases. The authors evaluate data diversity and utility by applying Synthline to defect classification in requirements specifications, comparing synthetic, real, and hybrid training configurations. Results show synthetic data alone can improve precision and recall, with hybrid training achieving up to 85% precision gains and 2x recall over real data alone, though diversity remains lower than real data and source curation is critical. The work demonstrates the practicality of PL-guided synthetic data to alleviate RE data scarcity and provides open datasets and code to support reproducibility and broader adoption, with plans to extend to more RE use cases and deeper evaluation.

Abstract

While modern Requirements Engineering (RE) heavily relies on natural language processing and Machine Learning (ML) techniques, their effectiveness is limited by the scarcity of high-quality datasets. This paper introduces Synthline, a Product Line (PL) approach that leverages Large Language Models to systematically generate synthetic RE data for classification-based use cases. Through an empirical evaluation conducted in the context of using ML for the identification of requirements specification defects, we investigated both the diversity of the generated data and its utility for training downstream models. Our analysis reveals that while synthetic datasets exhibit less diversity than real data, they are good enough to serve as viable training resources. Moreover, our evaluation shows that combining synthetic and real data leads to substantial performance improvements. Specifically, hybrid approaches achieve up to 85% improvement in precision and a 2x increase in recall compared to models trained exclusively on real data. These findings demonstrate the potential of PL-based synthetic data generation to address data scarcity in RE. We make both our implementation and generated datasets publicly available to support reproducibility and advancement in the field.

Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models

TL;DR

Synthline introduces a minimal Product Line approach to generate synthetic RE data using large language models, controlled by a Feature Model to support classification-based use cases. The authors evaluate data diversity and utility by applying Synthline to defect classification in requirements specifications, comparing synthetic, real, and hybrid training configurations. Results show synthetic data alone can improve precision and recall, with hybrid training achieving up to 85% precision gains and 2x recall over real data alone, though diversity remains lower than real data and source curation is critical. The work demonstrates the practicality of PL-guided synthetic data to alleviate RE data scarcity and provides open datasets and code to support reproducibility and broader adoption, with plans to extend to more RE use cases and deeper evaluation.

Abstract

While modern Requirements Engineering (RE) heavily relies on natural language processing and Machine Learning (ML) techniques, their effectiveness is limited by the scarcity of high-quality datasets. This paper introduces Synthline, a Product Line (PL) approach that leverages Large Language Models to systematically generate synthetic RE data for classification-based use cases. Through an empirical evaluation conducted in the context of using ML for the identification of requirements specification defects, we investigated both the diversity of the generated data and its utility for training downstream models. Our analysis reveals that while synthetic datasets exhibit less diversity than real data, they are good enough to serve as viable training resources. Moreover, our evaluation shows that combining synthetic and real data leads to substantial performance improvements. Specifically, hybrid approaches achieve up to 85% improvement in precision and a 2x increase in recall compared to models trained exclusively on real data. These findings demonstrate the potential of PL-based synthetic data generation to address data scarcity in RE. We make both our implementation and generated datasets publicly available to support reproducibility and advancement in the field.
Paper Structure (17 sections, 2 figures, 8 tables)

This paper contains 17 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Overview of the Synthline approach
  4. Domain Engineering
  5. Feature Model
  6. Synthline Architecture
  7. Application Engineering
  8. Identification of Requirements Specification Defects
  9. Data Generation
  10. Evaluation
  11. Diversity Evaluation
  12. Utility Evaluation
  13. Results
  14. RQ1: How does the diversity of the generated data compare to that of real data?
  15. RQ2: How does training on synthetic data affect model performance relative to training exclusively on real data?
  16. ...and 2 more sections

Figures (2)

  • Figure 1: High-level architecture of Synthline.
  • Figure 2: Distribution of cosine similarity of text pairs sampled from the same class.