Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval Augmented (Knowledge Graph), and Large Language Model-Driven Design Structure Matrix (DSM) Generation of Cyber-Physical Systems

H. Sinan Bank, Daniel R. Herber

TL;DR

This work investigates automated DSM generation for cyber-physical systems using augmented language approaches (RAG) and graph-grounded retrieval (GraphRAG), validated on two representative use cases (a power screwdriver and a CubeSat). By comparing baseline LLMs, RAG, and GraphRAG across component-identification and relationship-extraction tasks, the study reveals that model architecture and prompt/reference configuration often drive performance more than sheer size, with context-grounded approaches yielding benefits in specific configurations. It also uncovers that simply aggregating external references can harm performance, underscoring the need for careful reference curation and alignment strategies. The authors provide open-source code to support reproducibility and further validation, highlighting both the potential and current limitations of grounding automated DSM generation in verified domain knowledge for CPS architecture design.

Abstract

We explore the potential of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and Graph-based RAG (GraphRAG) for generating Design Structure Matrices (DSMs). We test these methods on two distinct use cases -- a power screwdriver and a CubeSat with known architectural references -- evaluating their performance on two key tasks: determining relationships between predefined components, and the more complex challenge of identifying components and their subsequent relationships. We measure the performance by assessing each element of the DSM and overall architecture. Despite design and computational challenges, we identify opportunities for automated DSM generation, with all code publicly available for reproducibility and further feedback from the domain experts.

Retrieval Augmented (Knowledge Graph), and Large Language Model-Driven Design Structure Matrix (DSM) Generation of Cyber-Physical Systems

TL;DR

This work investigates automated DSM generation for cyber-physical systems using augmented language approaches (RAG) and graph-grounded retrieval (GraphRAG), validated on two representative use cases (a power screwdriver and a CubeSat). By comparing baseline LLMs, RAG, and GraphRAG across component-identification and relationship-extraction tasks, the study reveals that model architecture and prompt/reference configuration often drive performance more than sheer size, with context-grounded approaches yielding benefits in specific configurations. It also uncovers that simply aggregating external references can harm performance, underscoring the need for careful reference curation and alignment strategies. The authors provide open-source code to support reproducibility and further validation, highlighting both the potential and current limitations of grounding automated DSM generation in verified domain knowledge for CPS architecture design.

Abstract

We explore the potential of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and Graph-based RAG (GraphRAG) for generating Design Structure Matrices (DSMs). We test these methods on two distinct use cases -- a power screwdriver and a CubeSat with known architectural references -- evaluating their performance on two key tasks: determining relationships between predefined components, and the more complex challenge of identifying components and their subsequent relationships. We measure the performance by assessing each element of the DSM and overall architecture. Despite design and computational challenges, we identify opportunities for automated DSM generation, with all code publicly available for reproducibility and further feedback from the domain experts.
Paper Structure (17 sections, 2 equations, 10 figures, 7 tables)

This paper contains 17 sections, 2 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. The Architectural Representation
  4. Automated Architecture Generation
  5. Graph-based Methods
  6. Language-Based Methods
  7. Method
  8. Large Language Models-LLMs
  9. Augmented Language Models-ALMs
  10. Use-case Studies
  11. Architectural Structure of Existing Systems
  12. Results and Discussion
  13. Evaluations of LLM, RAG, and GraphRAG for DSM Generation
  14. Conclusion and Future Work
  15. Appendix A
  16. ...and 2 more sections

Figures (10)

  • Figure 1: Architectural Representations of Sample Systems a) Steam Engine di2021evaluatingb) F1Tenth Autonomous Car von2024toward
  • Figure 2: Overview of a) Baseline LLM b) Baseline LLM with RAGlewis2020retrievalc) LLM with KG and RAGedge2024local
  • Figure 3: An Overview of Step 1 of the proposed approach
  • Figure 4: An Overview of Step 2 with associated repositorybankh2024xlm steps
  • Figure 5: An Overview of Step 3 - Analysis and Visualization
  • ...and 5 more figures