VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion

TL;DR

This work tackles the sparsity of safety-critical autonomous driving scenarios by introducing a three-layer framework in which a Vision-Language Model acts as a strategic planner to guide an adaptive diffusion model. The diffusion core learns real-world driving distributions while the tactical layer shapes adaptive guidance through time windows and trigger conditions, enabling closed-loop interactions with the vehicle under test. A multimodal VLM input pipeline plus an accident knowledge database and chain-of-thought prompting drive objective determination and guidance function formulation, yielding realistic, diverse, and highly interactive testing scenarios validated on nuPlan across four ADS algorithms. Results show substantial increases in safety-critical exposure and demonstrate robust adaptability to different AUTs and VLMs, with ablations confirming the value of multimodal inputs and CoT reasoning.

Abstract

The safe deployment of autonomous driving systems (ADSs) relies on comprehensive testing and evaluation. However, safety-critical scenarios that can effectively expose system vulnerabilities are extremely sparse in the real world. Existing scenario generation methods face challenges in efficiently constructing long-tail scenarios that ensure fidelity, criticality, and interactivity, while particularly lacking real-time dynamic response capabilities to the vehicle under test (VUT). To address these challenges, this paper proposes a safety-critical testing scenario generation framework that integrates the high-level semantic understanding capabilities of Vision Language Models (VLMs) with the fine-grained generation capabilities of adaptive guided diffusion models. The framework establishes a three-layer hierarchical architecture comprising a strategic layer for VLM-directed scenario generation objective determination, a tactical layer for guidance function formulation, and an operational layer for guided diffusion execution. We first establish a high-quality fundamental diffusion model that learns the data distribution of real driving scenarios. Next, we design an adaptive guided diffusion method that enables real-time, precise control of background vehicles (BVs) in closed-loop simulation. The VLM is then incorporated to autonomously generate scenario generation objectives and guidance functions through deep scenario understanding and risk reasoning, ultimately guiding the diffusion model to achieve VLM-directed scenario generation. Experimental results demonstrate that the proposed method can efficiently generate realistic, diverse, and highly interactive safety-critical testing scenarios. Furthermore, case studies validate the adaptability and VLM-directed generation performance of the proposed method.

Figures (9)

• Figure 1: Framework of the proposed method.
• Figure 2: Structure of the fundamental diffusion model.
• Figure 3: An example of the VLM input information.
• Figure 4: The prompts for each CoT reasoning step.
• Figure 5: The co-simulation platform.