Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concept-Guided LLM Agents for Human-AI Safety Codesign

Florian Geissler, Karsten Roscher, Mario Trapp

TL;DR

This work tackles the challenge of using LLMs for safety analysis in software systems by introducing a concept-guided hybrid framework that verbalizes a structured system model into an intermediate representation and guides LLM reasoning through a cascade of micro-decisions. The LLM agent, built with LangChain, interacts with a graph-based system model and external tools (via retrieval-augmented generation and deterministic functions) to perform fault propagation, critical-path, single-point-of-failure, and replication analyses. A key contribution is the graph verbalization and a predefined-concept decision network that constrains tool triggers, improving safety and reliability over naïve CoT approaches. Demonstrated on a simplified automated driving architecture, the method yields accurate task identification and safe graph updates, offering a scalable, interactive Human-AI safety codesign framework with potential applicability to larger safety-critical domains.

Abstract

Generative AI is increasingly important in software engineering, including safety engineering, where its use ensures that software does not cause harm to people. This also leads to high quality requirements for generative AI. Therefore, the simplistic use of Large Language Models (LLMs) alone will not meet these quality demands. It is crucial to develop more advanced and sophisticated approaches that can effectively address the complexities and safety concerns of software systems. Ultimately, humans must understand and take responsibility for the suggestions provided by generative AI to ensure system safety. To this end, we present an efficient, hybrid strategy to leverage LLMs for safety analysis and Human-AI codesign. In particular, we develop a customized LLM agent that uses elements of prompt engineering, heuristic reasoning, and retrieval-augmented generation to solve tasks associated with predefined safety concepts, in interaction with a system model graph. The reasoning is guided by a cascade of micro-decisions that help preserve structured information. We further suggest a graph verbalization which acts as an intermediate representation of the system model to facilitate LLM-graph interactions. Selected pairs of prompts and responses relevant for safety analytics illustrate our method for the use case of a simplified automated driving system.

Concept-Guided LLM Agents for Human-AI Safety Codesign

TL;DR

This work tackles the challenge of using LLMs for safety analysis in software systems by introducing a concept-guided hybrid framework that verbalizes a structured system model into an intermediate representation and guides LLM reasoning through a cascade of micro-decisions. The LLM agent, built with LangChain, interacts with a graph-based system model and external tools (via retrieval-augmented generation and deterministic functions) to perform fault propagation, critical-path, single-point-of-failure, and replication analyses. A key contribution is the graph verbalization and a predefined-concept decision network that constrains tool triggers, improving safety and reliability over naïve CoT approaches. Demonstrated on a simplified automated driving architecture, the method yields accurate task identification and safe graph updates, offering a scalable, interactive Human-AI safety codesign framework with potential applicability to larger safety-critical domains.

Abstract

Generative AI is increasingly important in software engineering, including safety engineering, where its use ensures that software does not cause harm to people. This also leads to high quality requirements for generative AI. Therefore, the simplistic use of Large Language Models (LLMs) alone will not meet these quality demands. It is crucial to develop more advanced and sophisticated approaches that can effectively address the complexities and safety concerns of software systems. Ultimately, humans must understand and take responsibility for the suggestions provided by generative AI to ensure system safety. To this end, we present an efficient, hybrid strategy to leverage LLMs for safety analysis and Human-AI codesign. In particular, we develop a customized LLM agent that uses elements of prompt engineering, heuristic reasoning, and retrieval-augmented generation to solve tasks associated with predefined safety concepts, in interaction with a system model graph. The reasoning is guided by a cascade of micro-decisions that help preserve structured information. We further suggest a graph verbalization which acts as an intermediate representation of the system model to facilitate LLM-graph interactions. Selected pairs of prompts and responses relevant for safety analytics illustrate our method for the use case of a simplified automated driving system.
Paper Structure (5 sections, 3 figures, 1 table)

This paper contains 5 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Model
  3. Experiments
  4. Conclusion and Outlook
  5. Acknowledgments

Figures (3)

  • Figure 1: Layout of the Human-AI safety codesign framework: A user interacts with the LLM agent through a chat prompt and receives a text response. The LLM agent interfaces to a database containing a graph description of the system model in IR, as well as safety concepts, and analysis tools. System manipulations can update the database and alter the system model. The system model and its changes are displayed to the user.
  • Figure 2: Outline of the workflow of LLM agent decisions and database interactions. The custom agent runs a cascading decision layer to identify the task type, and a subsequent layer to formulate the task for information retrieval. The latter looks up knowledge from a vectorstore database, or uses functional tools to calculate for example the critical path.
  • Figure 3: Example use case of a simplified automated driving system. The node labels denote the component name (top row) and the fault gate attribute below (if none is given, an AND fault gate of all inputs is assumed). $2OO3$ means that two out of three of the listed inputs are required. Start and end nodes are further labeled explicitly. The figure is a pydot visualization of the ECore file.