Spatial-aware Vision Language Model for Autonomous Driving

TL;DR

This work tackles the key limitation of Vision-Language Models in autonomous driving: lack of explicit 3D metric understanding. It introduces LVLDrive, a LiDAR–Vision–Language framework that linearly fuses LiDAR with image inputs via a Gradual Fusion Q-Former to preserve existing VLM priors while improving 3D spatial reasoning. To train and evaluate the enhanced spatial capabilities, the authors propose SA-QA, a spatially aware QA dataset grounded in 3D annotations from nuScenes, augmented with cross-modal prompts and a modality-masking mechanism. Extensive experiments on nuScenes-based benchmarks show LVLDrive improves scene understanding, metric spatial perception, and planning reliability over vision-only baselines, underscoring the importance of explicit 3D cues for trustworthy VLM-driven autonomous systems.

Abstract

While Vision-Language Models (VLMs) show significant promise for end-to-end autonomous driving by leveraging the common sense embedded in language models, their reliance on 2D image cues for complex scene understanding and decision-making presents a critical bottleneck for safety and reliability. Current image-based methods struggle with accurate metric spatial reasoning and geometric inference, leading to unreliable driving policies. To bridge this gap, we propose LVLDrive (LiDAR-Vision-Language), a novel framework specifically designed to upgrade existing VLMs with robust 3D metric spatial understanding for autonomous driving by incoperating LiDAR point cloud as an extra input modality. A key challenge lies in mitigating the catastrophic disturbance introduced by disparate 3D data to the pre-trained VLMs. To this end, we introduce a Gradual Fusion Q-Former that incrementally injects LiDAR features, ensuring the stability and preservation of the VLM's existing knowledge base. Furthermore, we develop a spatial-aware question-answering (SA-QA) dataset to explicitly teach the model advanced 3D perception and reasoning capabilities. Extensive experiments on driving benchmarks demonstrate that LVLDrive achieves superior performance compared to vision-only counterparts across scene understanding, metric spatial perception, and reliable driving decision-making. Our work highlights the necessity of explicit 3D metric data for building trustworthy VLM-based autonomous systems.

Figures (4)

• Figure 1: (a) Typical image-based VLMs take only images as input and train the LLM through image-conditioned question answering. In contrast, our LVLDrive leverages both image and LiDAR inputs and incorporates carefully designed spatial-aware QAs to encourage effective LiDAR integration and enhance spatial understanding.
• Figure 2: Overview of LVLDrive. LVLDrive takes text, images, and point clouds as multimodal inputs and employs three pretrained encoders, an LLM, and a Gradual Fusion Q-Former module that bridges visual and linguistic representations to generate task-specific responses. The fire symbol denotes trainable components.
• Figure 3: Gradual Fusion Q-Former. Each block contains two multi-head attention layers. The first layer uses learnable carrier and instance tokens as queries, keys, and values; the keys and values are extended with memory tokens when the memory bank is non-empty, and both queries and keys are augmented with 3D positional embeddings (3D PE) derived from reference points. The second layer introduces image and point features as keys and values, with queries and keys again augmented using their corresponding 3D positional embeddings. Note that the zero-initialized gate is applied only to point features. At the end of the module, the output queries are supervised by 3D perception objectives and LLM gradients separately.
• Figure 4: Examples to illustrate different question-answering pairs in SA-QA. The green dots, box, and line are highlighted only for visualization. The masked region and the red arrows are visible to the vision encoder.