Automated Concern Extraction from Textual Requirements of Cyber-Physical Systems: A Multi-solution Study

TL;DR

This work targets the gap between automated requirements concern extraction methods and real-world CPS practice by introducing ReqEBench, a public benchmark containing 2,721 requirements from 12 CPSs across multiple domains. It provides a two-tier taxonomy of concerns (entity and interaction), a rigorous multi-annotator ground truth, and a formal evaluation framework. A comprehensive comparative study shows that while current LLMs like GPT-4 outperform traditional methods, performance remains far from deployment-ready, with notable boundary and type errors and limited recall for certain entities. The authors analyze failure cases, propose directions such as retrieval-augmented prompts and domain-specific pretraining, and release replication resources to spur further advancements in CPS requirements concern extraction.

Abstract

Cyber-physical systems (CPSs) are characterized by a deep integration of the information space and the physical world, which makes the extraction of requirements concerns more challenging. Some automated solutions for requirements concern extraction have been proposed to alleviate the burden on requirements engineers. However, evaluating the effectiveness of these solutions, which relies on fair and comprehensive benchmarks, remains an open question. To address this gap, we propose ReqEBench, a new CPSs requirements concern extraction benchmark, which contains 2,721 requirements from 12 real-world CPSs. ReqEBench offers four advantages. It aligns with real-world CPSs requirements in multiple dimensions, e.g., scale and complexity. It covers comprehensive concerns related to CPSs requirements. It undergoes a rigorous annotation process. It covers multiple application domains of CPSs, e.g., aerospace and healthcare. We conducted a comparative study on three types of automated requirements concern extraction solutions and revealed their performance in real-world CPSs using our ReqEBench. We found that the highest F1 score of GPT-4 is only 0.24 in entity concern extraction. We further analyze failure cases of popular LLM-based solutions, summarize their shortcomings, and provide ideas for improving their capabilities. We believe ReqEBench will facilitate the evaluation and development of automated requirements concern extraction.

Figures (3)

• Figure 1: The Overview of ReqEBench Construction
• Figure 2: The overview of our comparative study
• Figure 3: The performance comparisons with different shot numbers.