SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving

TL;DR

SGDrive addresses the gaps of generalist Vision-Language Models in autonomous driving by introducing a scene-agent-goal hierarchical cognition framework that explicitly forecasts driving-relevant world knowledge. It integrates three layers of world understanding—scene geometry, safety-critical agents, and short-term goals—via 〈world〉 tokens and a block-wise attention scheme, and uses a diffusion-based planner to generate trajectories conditioned on this knowledge. The approach achieves state-of-the-art performance among camera-only methods on NAVSIM, with strong improvements in safety-focused metrics (e.g., No At-Fault Collisions and TTC) and robust ablations demonstrating the value of hierarchical world forecasting and structured attention. Overall, SGDrive demonstrates that forecasting structured future world states and disentangling high-level goals from low-level control can significantly enhance planning reliability and safety in real-world driving scenarios.

Abstract

Recent end-to-end autonomous driving approaches have leveraged Vision-Language Models (VLMs) to enhance planning capabilities in complex driving scenarios. However, VLMs are inherently trained as generalist models, lacking specialized understanding of driving-specific reasoning in 3D space and time. When applied to autonomous driving, these models struggle to establish structured spatial-temporal representations that capture geometric relationships, scene context, and motion patterns critical for safe trajectory planning. To address these limitations, we propose SGDrive, a novel framework that explicitly structures the VLM's representation learning around driving-specific knowledge hierarchies. Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition: drivers first perceive the overall environment (scene context), then attend to safety-critical agents and their behaviors, and finally formulate short-term goals before executing actions. This hierarchical decomposition provides the structured spatial-temporal representation that generalist VLMs lack, integrating multi-level information into a compact yet comprehensive format for trajectory planning. Extensive experiments on the NAVSIM benchmark demonstrate that SGDrive achieves state-of-the-art performance among camera-only methods on both PDMS and EPDMS, validating the effectiveness of hierarchical knowledge structuring for adapting generalist VLMs to autonomous driving.

Figures (9)

• Figure 1: (a) directly produces driving actions in textual form. (b) VLM generates action embeddings that decoded to produce the final trajectory. (c) Our SGDrive explicitly learns and forecasts scene, agent, and goal knowledge, providing structured driving-world understanding that strengthens action reasoning and improves generalization.
• Figure 2: The SGDrive pipeline introduces hierarchical 〈world〉 queries (scene, agent, and goal) for world modeling and trajectory generation. A key component is our world query encoder, which initializes these queries by integrating multi-modal priors from the ego state, historical trajectory, and visual features. These "prior-informed" queries are then processed by the VLM, alongside text and visual embeddings, to fuse all signals into a compact, hierarchical world representation.
• Figure 3: (a) Causal attention mask: input tokens are allowed to attend to tokens before. (b) Structure attention mask: prevents leakage by prohibiting all mutual attention between the different subquery sets (scene, agent, goal).
• Figure 4: Comparisons with state-of-the-art method on the Navtest benchmark.
• Figure 5: Qualitative visualization of our model's predictions (top row) versus the ground truth (bottom row). The visualization shows our model accurately forecasts these hierarchical states, which closely align with the ground truth.