Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Augmenting Scenario-Based Modeling with Generative AI

David Harel, Guy Katz, Assaf Marron, Smadar Szekely

TL;DR

The paper addresses the risk of inaccuracies when using generative AI for complex system modeling and proposes a safe, structured workflow that couples large language models with scenario-based modeling (SBM). It introduces a preamble-based prompting approach to constrain ChatGPT outputs to SBM transition systems and outlines an iterative methodology for evolving models with human and automated checks, including code generation, modeling, simulation, and SMT-like verification. Early results illustrate the approach's potential to produce executable SBM artifacts and to surface gaps, inconsistencies, and under-specifications. If validated in real-world case studies, this approach could enable safer, AI-assisted software engineering and more reliable automated analysis across development lifecycles.

Abstract

The manual modeling of complex systems is a daunting task; and although a plethora of methods exist that mitigate this issue, the problem remains very difficult. Recent advances in generative AI have allowed the creation of general-purpose chatbots, capable of assisting software engineers in various modeling tasks. However, these chatbots are often inaccurate, and an unstructured use thereof could result in erroneous system models. In this paper, we outline a method for the safer and more structured use of chatbots as part of the modeling process. To streamline this integration, we propose leveraging scenario-based modeling techniques, which are known to facilitate the automated analysis of models. We argue that through iterative invocations of the chatbot and the manual and automatic inspection of the resulting models, a more accurate system model can eventually be obtained. We describe favorable preliminary results, which highlight the potential of this approach.

On Augmenting Scenario-Based Modeling with Generative AI

TL;DR

The paper addresses the risk of inaccuracies when using generative AI for complex system modeling and proposes a safe, structured workflow that couples large language models with scenario-based modeling (SBM). It introduces a preamble-based prompting approach to constrain ChatGPT outputs to SBM transition systems and outlines an iterative methodology for evolving models with human and automated checks, including code generation, modeling, simulation, and SMT-like verification. Early results illustrate the approach's potential to produce executable SBM artifacts and to surface gaps, inconsistencies, and under-specifications. If validated in real-world case studies, this approach could enable safer, AI-assisted software engineering and more reliable automated analysis across development lifecycles.

Abstract

The manual modeling of complex systems is a daunting task; and although a plethora of methods exist that mitigate this issue, the problem remains very difficult. Recent advances in generative AI have allowed the creation of general-purpose chatbots, capable of assisting software engineers in various modeling tasks. However, these chatbots are often inaccurate, and an unstructured use thereof could result in erroneous system models. In this paper, we outline a method for the safer and more structured use of chatbots as part of the modeling process. To streamline this integration, we propose leveraging scenario-based modeling techniques, which are known to facilitate the automated analysis of models. We argue that through iterative invocations of the chatbot and the manual and automatic inspection of the resulting models, a more accurate system model can eventually be obtained. We describe favorable preliminary results, which highlight the potential of this approach.
Paper Structure (18 sections, 2 figures)

This paper contains 18 sections, 2 figures.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. Large Language Model-Based Chatbots
  4. Scenario-Based Modeling
  5. INTEGRATING CHATGPT AND SBM
  6. Basic Integration
  7. The Proposed Methodology
  8. USING THE METHOD IN THE DEVELOPMENT CYCLE
  9. Code Generation
  10. Modeling
  11. Play Out & Simulation
  12. SMT-Like Model Analysis
  13. Identifying Under-Specification
  14. Explainability
  15. Accommodating Semantic Flexibility
  16. ...and 3 more sections

Figures (2)

  • Figure 1: A scenario-based model for a system that controls the water level in a tank with hot and cold water taps (taken from HaKaMaWe14).
  • Figure 2: A simulation of scenario execution with ChatGPT. Top: textual description of scenarios in user prompts. Middle and bottom: The first two entries in ChatGPT's output when requested to play out the execution, showing triggered events, the scenarios that reacted to the triggered events, and the declarations of each scenario at the next synchronization point.