An Evaluation of Requirements Modeling for Cyber-Physical Systems via LLMs

TL;DR

The paper tackles the challenge of extracting and modeling CPS requirements from natural-language documents using problem diagrams. It introduces CPSBench, a real-world benchmark with 12 enterprise CPS documents and 30 tutorial cases, annotated for entities and interactions, and evaluates seven advanced LLMs under a few-shot prompting regime. Results show limited effectiveness (recall around 60%) and variability across entity types, with a notable gap in specialized CPS concepts; the study also presents a taxonomy of hallucinations in CPS requirements modeling. The work provides practical directions for improvement, including CPS-specific LLMs, retrieval-enhanced prompting, and collaborative model strategies, supported by open replication data.

Abstract

Cyber-physical systems (CPSs) integrate cyber and physical components and enable them to interact with each other to meet user needs. The needs for CPSs span rich application domains such as healthcare and medicine, smart home, smart building, etc. This indicates that CPSs are all about solving real-world problems. With the increasing abundance of sensing devices and effectors, the problems wanted to solve with CPSs are becoming more and more complex. It is also becoming increasingly difficult to extract and express CPS requirements accurately. Problem frame approach aims to shape real-world problems by capturing the characteristics and interconnections of components, where the problem diagram is central to expressing the requirements. CPSs requirements are generally presented in domain-specific documents that are normally expressed in natural language. There is currently no effective way to extract problem diagrams from natural language documents. CPSs requirements extraction and modeling are generally done manually, which is time-consuming, labor-intensive, and error-prone. Large language models (LLMs) have shown excellent performance in natural language understanding. It can be interesting to explore the abilities of LLMs to understand domain-specific documents and identify modeling elements, which this paper is working on. To achieve this goal, we first formulate two tasks (i.e., entity recognition and interaction extraction) and propose a benchmark called CPSBench. Based on this benchmark, extensive experiments are conducted to evaluate the abilities and limitations of seven advanced LLMs. We find some interesting insights. Finally, we establish a taxonomy of LLMs hallucinations in CPSs requirements modeling using problem diagrams. These results will inspire research on the use of LLMs for automated CPSs requirements modeling.

Figures (8)

• Figure 1: The process of requirements modeling for CPSs from natural language requirement document by developers
• Figure 2: The formulation of CPSs requirements modeling
• Figure 3: The Overview of CPSBench Construction
• Figure 4: An example of the prompt of entity recognition
• Figure 5: An example of the prompt of interaction extraction