A Multi-Agent LLM Framework for Design Space Exploration in Autonomous Driving Systems

TL;DR

This work tackles the challenge of efficiently exploring vast hardware/software design spaces for SAE Level-4 autonomous driving under diverse environments. It introduces a multi-agent LLM framework that integrates multi-modal reasoning with 3D simulation and profiling tools to automatically interpret execution outputs and guide design search. The approach demonstrates superior Pareto-front discovery and faster convergence compared with a genetic algorithm baseline in a robotaxi case study, validating the potential of LLM-driven design automation for autonomous driving systems. By coordinating specialized agents and leveraging both textual and visual performance data, the framework automates bottleneck detection and design-point prediction, enabling scalable, cost-efficient system design without human intervention.

Abstract

Designing autonomous driving systems requires efficient exploration of large hardware/software configuration spaces under diverse environmental conditions, e.g., with varying traffic, weather, and road layouts. Traditional design space exploration (DSE) approaches struggle with multi-modal execution outputs and complex performance trade-offs, and often require human involvement to assess correctness based on execution outputs. This paper presents a multi-agent, large language model (LLM)-based DSE framework, which integrates multi-modal reasoning with 3D simulation and profiling tools to automate the interpretation of execution outputs and guide the exploration of system designs. Specialized LLM agents are leveraged to handle user input interpretation, design point generation, execution orchestration, and analysis of both visual and textual execution outputs, which enables identification of potential bottlenecks without human intervention. A prototype implementation is developed and evaluated on a robotaxi case study (an SAE Level 4 autonomous driving application). Compared with a genetic algorithm baseline, the proposed framework identifies more Pareto-optimal, cost-efficient solutions with reduced navigation time under the same exploration budget. Experimental results also demonstrate the efficiency of the adoption of the LLM-based approach for DSE. We believe that this framework paves the way to the design automation of autonomous driving systems.

Figures (12)

• Figure 1: The architecture of the LLM-augmented DSE framework for the designs of autonomous driving systems.
• Figure 2: The user instruction example for the Interpretation Agent to start a DSE task.
• Figure 3: The example system prompt, $S$, of our case study.
• Figure 4: The example of dynamic instruction before a simulation, $I$, offering a few randomly-chosen design points to refer.
• Figure 5: The example dynamic instruction after a simulation, $I$, of our case study.