Vision-Language Cross-Attention for Real-Time Autonomous Driving

TL;DR

The paper addresses real-time autonomous driving by integrating perception, HD-map context, and goal reasoning within a single vision-language transformer. It introduces XYZ-Drive, which tokenizes RGB frames, 25m×25m BEV map patches, and waypoint text, applying a lightweight goal-centered cross-attention to fuse modalities before a partially fine-tuned LLaMA-3.2 11B Vision backbone for steering and speed. On the MD-NEX Outdoor-Driving benchmark, XYZ-Drive achieves SR=95% and SPL=0.80, outperforming the prior best PhysNav-DG by about 15 percentage points in SR and halving collisions, all with a single backbone and no separate explanation branch. The results suggest that early, token-level fusion of intent and map layout enables accurate, transparent, real-time driving and offers a scalable path toward semantics-aware vehicle control.

Abstract

Autonomous cars need geometric accuracy and semantic understanding to navigate complex environments, yet most stacks handle them separately. We present XYZ-Drive, a single vision-language model that reads a front-camera frame, a 25m $\times$ 25m overhead map, and the next waypoint, then outputs steering and speed. A lightweight goal-centered cross-attention layer lets waypoint tokens highlight relevant image and map patches, supporting both action and textual explanations, before the fused tokens enter a partially fine-tuned LLaMA-3.2 11B model. On the MD-NEX Outdoor-Driving benchmark XYZ-Drive attains 95% success and 0.80 Success weighted by Path Length (SPL), surpassing PhysNav-DG by 15%. and halving collisions, all while significantly improving efficiency by using only a single branch. Sixteen ablations explain the gains. Removing any modality (vision, waypoint, map) drops success by up to 11%, confirming their complementary roles and rich connections. Replacing goal-centered attention with simple concatenation cuts 3% in performance, showing query-based fusion injects map knowledge more effectively. Keeping the transformer frozen loses 5%, showing the importance of fine-tuning when applying VLMs for specific tasks such as autonomous driving. Coarsening map resolution from 10 cm to 40 cm blurs lane edges and raises crash rate. Overall, these results demonstrate that early, token-level fusion of intent and map layout enables accurate, transparent, real-time driving.