SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries

TL;DR

SAFE-SIM addresses the need for realistic, long-horizon safety-critical testing of autonomous vehicles by using diffusion-based trajectory generation with controllable adversaries in a closed-loop setting. It introduces guided diffusion and a novel partial diffusion framework that seeds trajectory proposals and allows fine-grained control over collision types and safety-criticality through objectives such as J_adv, J_v, and J_ttc, while maintaining realism via J_reg and route/Gaussian guidance. Empirical results on nuScenes and nuPlan show improved realism and controllability compared to baselines like STRIVE and DiffScene, with the ability to vary collision types and TTC-based safety characteristics across planners. The work demonstrates that diffusion models offer a robust, flexible foundation for interactive safety-critical traffic simulation, enabling safer, more thorough evaluation and potential policy training in real-world AV contexts.

Abstract

Evaluating the performance of autonomous vehicle planning algorithms necessitates simulating long-tail safety-critical traffic scenarios. However, traditional methods for generating such scenarios often fall short in terms of controllability and realism; they also neglect the dynamics of agent interactions. To address these limitations, we introduce SAFE-SIM, a novel diffusion-based controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We develop a novel approach to simulate safety-critical scenarios through an adversarial term in the denoising process of diffusion models, which allows an adversarial agent to challenge a planner with plausible maneuvers while all agents in the scene exhibit reactive and realistic behaviors. Furthermore, we propose novel guidance objectives and a partial diffusion process that enables users to control key aspects of the scenarios, such as the collision type and aggressiveness of the adversarial agent, while maintaining the realism of the behavior. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability. These findings affirm that diffusion models provide a robust and versatile foundation for safety-critical, interactive traffic simulation, extending their utility across the broader autonomous driving landscape. Project website: https://safe-sim.github.io/.

Figures (8)

• Figure 1: Overview of Safe-Sim Framework for Controllable Safety-Critical Closed-Loop Simulation. This framework evaluates a planner within scenarios featuring multiple controllable reactive agents. These agents have two distinct roles: adversarial agents, which actively challenge the planner by exhibiting controllable adversarial behaviors such as specific collision types and levels of aggressiveness, and non-adversarial agents, which follow normal driving behavior to maintain the realism of the entire scene. Such a setup facilitates the generation of various realistic, interactive, and safety-critical scenarios, providing a thorough evaluation of the planner’s capabilities.
• Figure 2: Guided Diffusion Process for the Adversarial Agent. This process optimizes the adversarial agent's trajectory using the adversarial cost function $J_{\text{adv}}$ to the ego vehicle. In particular, we introduce $J_{\text{control}}$ to vary the adversarial behavior. Simultaneously, it applies regularization through $J_{\text{reg}}$ for maintaining realism.
• Figure 3: Framework for Partial Diffusion. We generate proposals based on domain knowledge (e.g., collision types). Users can adjust noise levels to balance between user control and the model’s data distribution.
• Figure 4: Partial Diffusion Results of Rule-Based Planner on NuScenes Dataset. The safety-critical scenarios show the framework's ability to create realistic and challenging situations, varying collision types based on different trajectory proposals. The gradient lines reflects the planned trajectories in the next 3.2 seconds.
• Figure A1: Illustration of Diverse Collision Scenarios via Partial Diffusion. This figure showcases example simulations that highlight how varying trajectory proposals can influence the occurrence and type of collisions. The black line represents the trajectory proposals for the adversarial vehicle.