ShaRE your Data! Characterizing Datasets for LLM-based Requirements Engineering

TL;DR

The paper addresses the data shortage in RE for LLM-based research by systematically mapping public datasets used in LLM4RE tasks. It identifies 62 publicly available RE datasets across 43 primary studies and characterizes them along multiple attributes, including artifact type, granularity, RE stage, task, domain, language, and size, compiling a public catalogue. The authors reveal gaps such as limited elicitation and management data, English-dominant and software-centric domains, and small dataset scales, and propose a standardized, extensible characterization framework. This work provides a foundation for improved dataset selection, reuse, and benchmarking in NLP4RE and LLM4RE, and outlines a roadmap to broaden coverage and integrate with open repositories.

Abstract

[Context] Large Language Models (LLMs) rely on domain-specific datasets to achieve robust performance across training and inference stages. However, in Requirements Engineering (RE), data scarcity remains a persistent limitation reported in surveys and mapping studies. [Question/Problem] Although there are multiple datasets supporting LLM-based RE tasks (LLM4RE), they are fragmented and poorly characterized, limiting reuse and comparability. This research addresses the limited visibility and characterization of datasets used in LLM4RE. We investigate which public datasets are employed, how they can be systematically characterized, and which RE tasks and dataset descriptors remain under-represented. [Ideas/Results] To address this, we conduct a systematic mapping study to identify and analyse datasets used in LLM4RE research. A total of 62 publicly available datasets are referenced across 43 primary studies. Each dataset is characterized along descriptors such as artifact type, granularity, RE stage, task, domain, and language. Preliminary findings show multiple research gaps, including limited coverage for elicitation tasks, scarce datasets for management activities beyond traceability, and limited multilingual availability. [Contribution] This research preview offers a public catalogue and structured characterization scheme to support dataset selection, comparison, and reuse in LLM4RE research. Future work will extend the scope to grey literature, as well as integration with open dataset and benchmark repositories.

Figures (3)

• Figure 1: Distribution of datasets across metadata attributes. Labels with fewer than three occurrences are omitted to improve readability.
• Figure 2: Usage growth in the number of datasets over time.
• Figure 3: Distribution of datasets across RE activities and tasks.