Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are requirements really all you need? A case study of LLM-driven configuration code generation for automotive simulations

Krzysztof Lebioda, Nenad Petrovic, Fengjunjie Pan, Vahid Zolfaghari, Andre Schamschurko, Alois Knoll

TL;DR

The paper investigates translating high-level automotive requirements into executable CARLA configuration code using an end-to-end LLM-driven pipeline. It decomposes the task into vehicle-definition, pre-conditions, and post-conditions modules, plus a merger step, and evaluates prompting strategies to understand their impact on translation quality. Results show strong performance for direct requirements but notable challenges with abstract ones, with improvements achieved through logical grouping and reasoning prompts, especially Chain-of-Thought. The work offers a practical pathway for rapid prototyping in software-defined automotive testing and points to future directions such as DSLs, RAG-based extraction, and event-chain-driven code generation to broaden applicability.

Abstract

Large Language Models (LLMs) are taking many industries by storm. They possess impressive reasoning capabilities and are capable of handling complex problems, as shown by their steadily improving scores on coding and mathematical benchmarks. However, are the models currently available truly capable of addressing real-world challenges, such as those found in the automotive industry? How well can they understand high-level, abstract instructions? Can they translate these instructions directly into functional code, or do they still need help and supervision? In this work, we put one of the current state-of-the-art models to the test. We evaluate its performance in the task of translating abstract requirements, extracted from automotive standards and documents, into configuration code for CARLA simulations.

Are requirements really all you need? A case study of LLM-driven configuration code generation for automotive simulations

TL;DR

The paper investigates translating high-level automotive requirements into executable CARLA configuration code using an end-to-end LLM-driven pipeline. It decomposes the task into vehicle-definition, pre-conditions, and post-conditions modules, plus a merger step, and evaluates prompting strategies to understand their impact on translation quality. Results show strong performance for direct requirements but notable challenges with abstract ones, with improvements achieved through logical grouping and reasoning prompts, especially Chain-of-Thought. The work offers a practical pathway for rapid prototyping in software-defined automotive testing and points to future directions such as DSLs, RAG-based extraction, and event-chain-driven code generation to broaden applicability.

Abstract

Large Language Models (LLMs) are taking many industries by storm. They possess impressive reasoning capabilities and are capable of handling complex problems, as shown by their steadily improving scores on coding and mathematical benchmarks. However, are the models currently available truly capable of addressing real-world challenges, such as those found in the automotive industry? How well can they understand high-level, abstract instructions? Can they translate these instructions directly into functional code, or do they still need help and supervision? In this work, we put one of the current state-of-the-art models to the test. We evaluate its performance in the task of translating abstract requirements, extracted from automotive standards and documents, into configuration code for CARLA simulations.
Paper Structure (23 sections, 5 figures, 5 tables)

This paper contains 23 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and related works
  3. Methods
  4. Requirement types
  5. Vehicle definition pipeline
  6. Pre-conditions pipeline
  7. Post-conditions pipeline
  8. Merger
  9. Prompt types
  10. Evaluation methods
  11. Experiments and result analysis
  12. Vehicle definition - automatic generation of sensor configuration
  13. Vehicle definition - prompting style
  14. Vehicle definition - requirement order
  15. Vehicle definition - performance as function of number of requirements
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Overview of the test setup. Simulation pre-conditions (purple) refer to the setup of the scene - agent positions, their desired behavior, weather conditions, etc. Simulation post-conditions (green) reflect the desired outcomes of the test and are largely related to the vehicle's telemetry. Vehicle definition (orange) is responsible for specifying the ego vehicle's sensor array. All modules in the system under test are considered to be fixed. The requirements visible in the figure are simplified.
  • Figure 2: Overview of the generative system.
  • Figure :
  • Figure :
  • Figure :