GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

TL;DR

This work presents GraphPilot, a model-agnostic approach to grounded planning in language-based autonomous driving by injecting serialized traffic scene graphs into prompts. By exploring scene-graph abstractions, serialization formats, and prompt templates, the study demonstrates that training-time relational supervision yields substantial and persistent driving improvements, even when test-time graphs are not provided. The key finding is that lean, actor-only graphs can achieve strong performance with efficient prompts, enabling practical deployment without the overhead of real-time graph generation. The results on the LangAuto benchmark show notable gains for state-of-the-art systems, highlighting the value of explicit relational priors for structured decision making in autonomous driving.

Abstract

Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on topology-aware reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes these relational dependencies, limiting their ability to infer how agents and other traffic entities influence one another from raw sensor data. In this work, we bridge this gap with a novel model-agnostic method that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at various abstraction levels and formats, and incorporate them into the models via structured prompt templates, enabling a systematic analysis of when and how relational supervision is most beneficial. Extensive evaluations on the public LangAuto benchmark show that scene graph conditioning of state-of-the-art approaches yields large and persistent improvement in driving performance. Notably, we observe up to a 15.6\% increase in driving score for LMDrive and 17.5\% for BEVDriver, indicating that models can better internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.

Figures (4)

• Figure 1: Explicit relational grounding through scene graph conditioning. We visualize four setups, where two binary digits indicate scene graph usage during training and testing: SG00-PT (baseline pretrained, no scene graphs), SG00-FT (baseline fine-tuned without scene graphs), SG10 (scene graphs only during training), and SG11 (scene graphs during training and testing). Models trained under SG10 perform largely on par with SG11, suggesting that they internalize relational structure during training. Performance indicates (mean) driving scores, cf. \ref{['tab:lmdrive_bevdriver_ckpt_ft']} and \ref{['tab:sg_augmented_ft']}.
• Figure 2: Scene graph construction. Each traffic scene is represented as a structured, labeled graph capturing entities (actors, objects, structure) and their relations. We define three levels of abstraction: Full (all node types and relations), Road-Level (collapsed structural detail), and Actor-Only (actors and their pairwise interactions), enabling analysis of the trade-off between relational fidelity and prompt efficiency. Dashed nodes and edges indicate optional elements that are not always present in a given scene.
• Figure 3: Serialization formats. We serialize scene graphs as Text, JSON, or YAML, each encoding subject-predicate-object triplets. Text uses compact natural-language packing for brevity, JSON provides a structured and parser-friendly representation, and YAML achieves lower token counts through minimal syntax.
• Figure 4: Prompt templates. Three prompt templates combine scene graphs with navigation commands: V1 uses direct concatenation, V2 adds ego-role framing and section headers, and V3 introduces a structured preamble with markdown-style fencing for consistent formatting.