ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving

TL;DR

ColaVLA tackles the brittleness and latency of vision–language model–based driving by relocating reasoning from textual chains to a unified vision–language–action latent space. It introduces a Cognitive Latent Reasoner that compresses scene understanding into compact meta-action priors via ego-adaptive token routing and a meta-action bank, followed by a Hierarchical Parallel Planner that decodes multi-scale trajectories in a single forward pass with a causality-preserving attention scheme. The approach achieves state-of-the-art open-loop and closed-loop performance on nuScenes, with significantly lower latency than autoregressive VLM methods, and demonstrates strong robustness through ablations and qualitative analyses. By bridging VLM cognition with continuous control in a latent, efficient framework, ColaVLA offers a scalable path to reliable, real-time autonomous driving with preserved interpretability.

Abstract

Autonomous driving requires generating safe and reliable trajectories from complex multimodal inputs. Traditional modular pipelines separate perception, prediction, and planning, while recent end-to-end (E2E) systems learn them jointly. Vision-language models (VLMs) further enrich this paradigm by introducing cross-modal priors and commonsense reasoning, yet current VLM-based planners face three key challenges: (i) a mismatch between discrete text reasoning and continuous control, (ii) high latency from autoregressive chain-of-thought decoding, and (iii) inefficient or non-causal planners that limit real-time deployment. We propose ColaVLA, a unified vision-language-action framework that transfers reasoning from text to a unified latent space and couples it with a hierarchical, parallel trajectory decoder. The Cognitive Latent Reasoner compresses scene understanding into compact, decision-oriented meta-action embeddings through ego-adaptive selection and only two VLM forward passes. The Hierarchical Parallel Planner then generates multi-scale, causality-consistent trajectories in a single forward pass. Together, these components preserve the generalization and interpretability of VLMs while enabling efficient, accurate and safe trajectory generation. Experiments on the nuScenes benchmark show that ColaVLA achieves state-of-the-art performance in both open-loop and closed-loop settings with favorable efficiency and robustness.

Figures (4)

• Figure 1: Illustration of inference paradigms. (a) Previous driving VLMs use text-based chain-of-thought, autoregressively emitting intermediate texts for sub-tasks; repeated decoding increases token cost and error compounding, causing high latency. (b) Our model performs latent reasoning in a VLA space with three forward passes, i.e. scene understanding, latent rethink, and parallel action decoding, removing autoregressive text and cutting inference latency while preserving decision-level interpretability.
• Figure 2: Overview of the ColaVLA framework. Multi-view image sequences are first processed by an image backbone and a Q-Former to perceive 3D objects and vectorized maps, producing visual tokens for subsequent reasoning and planning. On the left, the Cognitive Latent Reasoning module performs implicit reasoning through four stages, i.e.Understand, Recognize, Rethink, and Decide, to derive a driving strategy. On the right, the derived strategy selects corresponding meta-action queries from action bank, which are then transformed to multi-scale targets. These targets, together with the pruned context are fed into a Hierarchical Parallel Planner for one-pass, parallel trajectory decoding. The resulting multi-scale trajectories enable efficient, causal, and interpretable end-to-end planning.
• Figure 3: Causality-Preserving Hybrid Mask. Our mask is designed for the multi-scale targets within our planner. It enables information flow from the pruned context to all temporal scales, while maintaining temporal causality between adjacent scales.
• Figure 4: Qualitative visualization of multi-scale trajectory predictions. Red, yellow, and purple curves denote endpoint-only to full-trajectory predictions, while the green curve is the ground-truth. Right: BEV visualization with ego vehicle, agents, and trajectories.