LatentVLA: Efficient Vision-Language Models for Autonomous Driving via Latent Action Prediction

TL;DR

LatentVLA tackles the limitations of Vision-Language-Action models in autonomous driving by introducing ego-centric latent action learning via self-supervision, removing the need for language annotations, and distilling VLM knowledge into efficient vision-based planners. The method combines a latent action codebook with a two-stage training pipeline, fuses VLM embeddings with BEV features through either Transfuser- or ProFormer-based architectures, and uses a planning transformer to distill knowledge for real-time inference. It achieves state-of-the-art NAVSIM performance (PDMS up to 92.4) and strong zero-shot NuScenes generalization (average L2 $0.33$m), while delivering substantial speedups over vanilla VLM-based approaches through distillation. The results demonstrate improved generalization, reduced linguistic bias, and practical feasibility for deploying VLM-assisted driving systems in real-world settings.

Abstract

End-to-end autonomous driving models trained on largescale datasets perform well in common scenarios but struggle with rare, long-tail situations due to limited scenario diversity. Recent Vision-Language-Action (VLA) models leverage broad knowledge from pre-trained visionlanguage models to address this limitation, yet face critical challenges: (1) numerical imprecision in trajectory prediction due to discrete tokenization, (2) heavy reliance on language annotations that introduce linguistic bias and annotation burden, and (3) computational inefficiency from multi-step chain-of-thought reasoning hinders real-time deployment. We propose LatentVLA, a novel framework that employs self-supervised latent action prediction to train VLA models without language annotations, eliminating linguistic bias while learning rich driving representations from unlabeled trajectory data. Through knowledge distillation, LatentVLA transfers the generalization capabilities of VLA models to efficient vision-based networks, achieving both robust performance and real-time efficiency. LatentVLA establishes a new state-of-the-art on the NAVSIM benchmark with a PDMS score of 92.4 and demonstrates strong zeroshot generalization on the nuScenes benchmark.

Figures (4)

• Figure 1: LatentVLA Training Pipeline. The training of LatentVLA occurs in two main stages. In the first stage, presented on the left, we utilize ego-centric latent action tokens generated by the trained Latent Action Model as supervision to train the VLM for predicting latent actions. In the second stage, presented on the right, we freeze the parameters of the VLM and combine the visual embeddings and action embeddings obtained from the VLM using multi-head attention pooling. These embeddings are then integrated with traditional end-to-end methods and trained jointly.
• Figure 2: Architecture of LatentVLA integration with conventional end-to-end methods. We fuse VLM features with BEV representations through a dedicated fusion layer. (Top) LatentVLA based on TransFuser: VLM embeddings are integrated into the trajectory planning module via cross-attention while preserving auxiliary task features. (Bottom) LatentVLA based on iPad: VLM features are fused with the ProFormer module for iterative trajectory refinement.
• Figure 3: Distilling LatentVLA through Planning Transformer. The proposed pipeline minimizes both action prediction and feature distribution losses to transfer knowledge from LatentVLA, facilitating efficient trajectory planning.
• Figure 4: Qualitative comparison on challenging navtest scenarios. Top: In the roundabout, baseline Transfuser plans beyond the drivable area, while our methods generate smooth trajectories following the lane structure. Bottom: At the intersection, baseline enters the oncoming lane, whereas our LatentVLA maintains correct direction similar to ground truth. The distilled variant achieves comparable performance.