SG-CADVLM: A Context-Aware Decoding Powered Vision Language Model for Safety-Critical Scenario Generation

TL;DR

This work tackles the scarcity of safety-critical autonomous driving scenarios by transforming crash reports into executable simulations. It introduces SG-CADVLM, a context-aware, multimodal framework that couples Context-Aware Decoding with RAG to faithfully reconstruct road geometries and vehicle trajectories from crash descriptions and road diagrams. Key contributions include CAD-based alignment to crash context, multi-modal fusion for geometric fidelity, and RAG to ensure API-compliant, executable outputs, yielding a reported 469% rise in critical-risk scenario generation and substantial improvements in safety metrics. The approach enables more rigorous AV safety validation by linking real-world crash dynamics to simulation testing, though it notes computational cost and gaps like traffic signs in scenes as areas for future work.

Abstract

Autonomous vehicle safety validation requires testing on safety-critical scenarios, but these events are rare in real-world driving and costly to test due to collision risks. Crash reports provide authentic specifications of safety-critical events, offering a vital alternative to scarce real-world collision trajectory data. This makes them valuable sources for generating realistic high-risk scenarios through simulation. Existing approaches face significant limitations because data-driven methods lack diversity due to their reliance on existing latent distributions, whereas adversarial methods often produce unrealistic scenarios lacking physical fidelity. Large Language Model (LLM) and Vision Language Model (VLM)-based methods show significant promise. However, they suffer from context suppression issues where internal parametric knowledge overrides crash specifications, producing scenarios that deviate from actual accident characteristics. This paper presents SG-CADVLM (A Context-Aware Decoding Powered Vision Language Model for Safety-Critical Scenario Generation), a framework that integrates Context-Aware Decoding with multi-modal input processing to generate safety-critical scenarios from crash reports and road network diagrams. The framework mitigates VLM hallucination issues while enabling the simultaneous generation of road geometry and vehicle trajectories. The experimental results demonstrate that SG-CADVLM generates critical risk scenarios at a rate of 84.4% compared to 12.5% for the baseline methods, representing an improvement of 469%, while producing executable simulations for autonomous vehicle testing.

Figures (6)

• Figure 1: Histogram of FDE Distribution for Trajectory Dataset
• Figure 2: Framework Overview
• Figure 3: Input–output representation and processing pipeline of SG-CADVLM.
• Figure 4: The illustration of CAD mechanism
• Figure 5: Comparison of road network generation methods.