Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future

TL;DR

The paper surveys Vision-Language-Action (VLA) frameworks for autonomous driving, contrasting them with traditionalVision-Action (VA) pipelines. It introduces a two-pronged taxonomy—End-to-End VLA and Dual-System VLA—and further distinguishes textual versus numerical action generators and explicit versus implicit guidance. It synthesizes datasets, benchmarks, and metrics for evaluating VLA systems, and discusses key challenges such as robustness, interpretability, and instruction fidelity. The work outlines future directions including unified vision-language-world models, richer multimodal fusion, and safer deployment ecosystems, highlighting VLA’s potential to deliver interpretable, instruction-following, and human-aligned driving policies. Overall, VLA represents a promising path toward more capable and trustworthy autonomous driving agents, provided advancements in efficiency, grounding, and evaluation are achieved.

Abstract

Autonomous driving has long relied on modular "Perception-Decision-Action" pipelines, where hand-crafted interfaces and rule-based components often break down in complex or long-tailed scenarios. Their cascaded design further propagates perception errors, degrading downstream planning and control. Vision-Action (VA) models address some limitations by learning direct mappings from visual inputs to actions, but they remain opaque, sensitive to distribution shifts, and lack structured reasoning or instruction-following capabilities. Recent progress in Large Language Models (LLMs) and multimodal learning has motivated the emergence of Vision-Language-Action (VLA) frameworks, which integrate perception with language-grounded decision making. By unifying visual understanding, linguistic reasoning, and actionable outputs, VLAs offer a pathway toward more interpretable, generalizable, and human-aligned driving policies. This work provides a structured characterization of the emerging VLA landscape for autonomous driving. We trace the evolution from early VA approaches to modern VLA frameworks and organize existing methods into two principal paradigms: End-to-End VLA, which integrates perception, reasoning, and planning within a single model, and Dual-System VLA, which separates slow deliberation (via VLMs) from fast, safety-critical execution (via planners). Within these paradigms, we further distinguish subclasses such as textual vs. numerical action generators and explicit vs. implicit guidance mechanisms. We also summarize representative datasets and benchmarks for evaluating VLA-based driving systems and highlight key challenges and open directions, including robustness, interpretability, and instruction fidelity. Overall, this work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.

Figures (7)

• Figure 1: Outline. This work aims to provide a structured roadmap of the VLA paradigm for autonomous driving. We begin with Preliminary Foundations (Section \ref{['sec:preliminary']}), which formalize the general formulation of VLA models and detail their three core components: the multi-modal input modalities, the VLM backbone, and the action prediction head. It then traces the evolution from VA Models (Section \ref{['sec:va']}), which directly map perception to control, towards VLA Models (Section \ref{['sec:vla']}), which incorporate language-grounded reasoning and interpretable decision-making. We further categorize VLA architectures into two major paradigms -- End-to-End VLA (Section \ref{['sec:e2evla']}) and Dual-System VLA (Section \ref{['sec:dsvla']}) -- that differ in their integration of vision, language, and action modules. Next, we review Datasets & Benchmarks (Section \ref{['sec:datasets_benchmark_study']}) that enable both open-loop and closed-loop evaluation of driving intelligence. Finally, we discuss Challenges & Future Directions (Section \ref{['sec:challenges']}), highlighting interpretability, reasoning, and human-AI interaction as central themes driving the next generation of VLA-based autonomous driving research.
• Figure 2: Summary of representative VA and VLA models from existing literature, spanning End-to-End Models, World Models, Dual-Systems, etc. For the complete list of related approaches and the discussions on their specifications, configurations, and technical details, kindly refer to Section \ref{['sec:va']} and Section \ref{['sec:vla']}, respectively.
• Figure 3: The categorization of End-to-End VA models based on model structures and outputs, including Action-Only Models (Sec. \ref{['sec:actiononly']}), and Perception-Action Models (Sec. \ref{['sec:perceptionaction']}).
• Figure 4: The categorization of World Models based on prediction modalities, including Image-Based Models (Sec. \ref{['sec:imageworldmodel']}), Occupancy-Based Models (Sec. \ref{['sec:occworldmodel']}), and Latent-Based Models (Sec. \ref{['sec:latentworldmodel']}).
• Figure 5: The categorization of End-to-End VLA models based on the form of model outputs, including Textual Action Models (Sec. \ref{['sec:textualaction']}), and Numerical Action Models (Sec. \ref{['sec:numericalaction']}).