SAFE: Harnessing LLM for Scenario-Driven ADS Testing from Multimodal Crash Data

TL;DR

SAFE presents a multimodal, LLM-driven framework for scenario-based ADS testing that reconstructs crash-driven test scenarios from both crash sketches and summaries. By integrating a retrieval-augmented prompt generator, a compact domain-specific language, and a self-validation loop with Chain-of-Thought reasoning, SAFE reduces hallucinations and improves the fidelity of road networks, actors, and environments. In experiments on 50 crash cases across MetaDrive and BeamNG, SAFE outperforms LCTGen and AC3R in scenario extraction accuracy, scenario alignment, and ADS bug detection and crash reproduction, with statistically significant gains. The work demonstrates how multimodal data and knowledge-grounded prompting can scale realistic scenario synthesis for safety testing and provides a reusable, extensible framework for future ADS evaluation across diverse simulators and crash datasets.

Abstract

Ensuring the safety of Autonomous Driving Systems (ADS) requires realistic and reproducible test scenarios, yet extracting such scenarios from multimodal crash reports remains a major challenge. Large Language Models (LLMs) often hallucinate and lose map structure, resulting in unrealistic road layouts and vehicle behaviors. To address this, we introduce SAFE, a novel Scenario-based ADS testing Framework via multimodal Extraction, which leverages Retrieval-Augmented Generation (RAG), knowledge-grounded prompting, Chain-of-Thought (CoT) reasoning, and self-validation to improve scenario reconstruction from multimodal crash data. SAFE achieves 93.8% accuracy in extracting road network details, 80.0% for actor information, and 100% for environmental context. In human studies, SAFE outperforms LCTGen and AC3R in reconstructing consistent road networks and vehicle behaviors. Under identical ADS and simulator settings, SAFE detects 39 and 71 more safety violations than LCTGen and AC3R, respectively, and reproduces 12 more real-world crash cases than LCTGen. On 19 cases supported by AC3R, SAFE reproduces one additional crash case with statistically significant gains across five runs. It generates scenarios within 25 seconds and triggers violations after just 1 case (IDM) and 3 cases (PPO) in MetaDrive, as well as 1 case (Auto) in BeamNG. Code: https://github.com/Siwei-Luo-MQ/SAFE-ADS-Testing

Figures (9)

• Figure 1: SAFE Structure
• Figure 2: Straight Road Network in MetaDrive
• Figure 3: SAFE vs. LCTGen Human Study Results
• Figure 4: SAFE vs. AC3R Human Study Results
• Figure 5: SAFE vs. LCTGen on Scenario Construction