LinguaSim: Interactive Multi-Vehicle Testing Scenario Generation via Natural Language Instruction Based on Large Language Models

TL;DR

LinguaSim addresses the challenge of converting natural-language instructions into realistic, interactive 3D autonomous-vehicle scenarios while preserving adherence to user intent. It introduces a four-layer pipeline with dedicated LLM agents and a refinement loop that constrains adversarial behaviors through description guidance and ego-vehicle models. The framework achieves varying scenario criticalities (ACT from $0.072$ s to $3.532$ s) and significantly reduces crash rate after refinement (from $46.9\%$ to $6.3\%$), demonstrating improved alignment with user intent and realism. This approach enhances safety testing and training by enabling high-fidelity, linguistically specified scenarios without real-world data collection, with potential for broader benchmarking and cross-environment validation.

Abstract

The generation of testing and training scenarios for autonomous vehicles has drawn significant attention. While Large Language Models (LLMs) have enabled new scenario generation methods, current methods struggle to balance command adherence accuracy with the realism of real-world driving environments. To reduce scenario description complexity, these methods often compromise realism by limiting scenarios to 2D, or open-loop simulations where background vehicles follow predefined, non-interactive behaviors. We propose LinguaSim, an LLM-based framework that converts natural language into realistic, interactive 3D scenarios, ensuring both dynamic vehicle interactions and faithful alignment between the input descriptions and the generated scenarios. A feedback calibration module further refines the generation precision, improving fidelity to user intent. By bridging the gap between natural language and closed-loop, interactive simulations, LinguaSim constrains adversarial vehicle behaviors using both the scenario description and the autonomous driving model guiding them. This framework facilitates the creation of high-fidelity scenarios that enhance safety testing and training. Experiments show LinguaSim can generate scenarios with varying criticality aligned with different natural language descriptions (ACT: 0.072 s for dangerous vs. 3.532 s for safe descriptions; comfortability: 0.654 vs. 0.764), and its refinement module effectively reduces excessive aggressiveness in LinguaSim's initial outputs, lowering the crash rate from 46.9% to 6.3% to better match user intentions.

Figures (8)

• Figure 1: LinguaSim: a scenario generation framework based on LLM for autonomous vehicle testing and training
• Figure 2: A Overview of The Modular Design of LinguaSim
• Figure 3: The basic workflow of module Action Generator
• Figure 4: An example of the Behavior Topology Web generated by the Action Generator
• Figure 5: The Basic Pipeline of Refiner and Refine Commander