Safety-Critical Traffic Simulation with Guided Latent Diffusion Model

TL;DR

This paper addresses the challenge of evaluating autonomous driving systems under rare, safety-critical scenarios by generating realistic yet adversarial traffic situations. It introduces a guided latent diffusion model that operates in a graph-based VAE latent space, using DDIM-inspired sampling, task-specific guidance objectives, and a physical feasibility-based selection to ensure plausibility. The approach demonstrates superior adversarial effectiveness, realism, diversity, and generation efficiency on the nuScenes dataset compared with baselines like AdvSim and Strive. This scalable framework provides a practical, robust tool for safety validation of autonomous vehicles and can inform improvements in AV robustness and testing pipelines.

Abstract

Safety-critical traffic simulation plays a crucial role in evaluating autonomous driving systems under rare and challenging scenarios. However, existing approaches often generate unrealistic scenarios due to insufficient consideration of physical plausibility and suffer from low generation efficiency. To address these limitations, we propose a guided latent diffusion model (LDM) capable of generating physically realistic and adversarial safety-critical traffic scenarios. Specifically, our model employs a graph-based variational autoencoder (VAE) to learn a compact latent space that captures complex multi-agent interactions while improving computational efficiency. Within this latent space, the diffusion model performs the denoising process to produce realistic trajectories. To enable controllable and adversarial scenario generation, we introduce novel guidance objectives that drive the diffusion process toward producing adversarial and behaviorally realistic driving behaviors. Furthermore, we develop a sample selection module based on physical feasibility checks to further enhance the physical plausibility of the generated scenarios. Extensive experiments on the nuScenes dataset demonstrate that our method achieves superior adversarial effectiveness and generation efficiency compared to existing baselines while maintaining a high level of realism. Our work provides an effective tool for realistic safety-critical scenario simulation, paving the way for more robust evaluation of autonomous driving systems.

Figures (3)

• Figure 1: Overview of our guided LDM for safety-critical traffic simulation. Our model generates realistic yet adversarial driving scenarios by leveraging an LDM guided by adversarial objectives to effectively test AV systems.
• Figure 2: Overall framework of our proposed guided LDM. Graph-based VAEs encode posterior and prior scene input into latent representations, which are denoised by a U-Net conditioned on the prior latent. During the sampling stage, we introduce our proposed guidance objectives to guide the diffusion process toward generating adversarial safety-critical driving scenarios.
• Figure 3: Ablation study on guidance components. Reply refers to simulation using replayed non-ego vehicle trajectories from the dataset. W/O Guidance denotes our diffusion-based model without guidance, and W/ Guidance represents our proposed guidance-based diffusion model.