SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planning

TL;DR

SparseOccVLA tackles the integration of Vision-Language Models with semantic occupancy for unified 4D scene understanding and planning in autonomous driving. It introduces a Sparse Occupancy Encoder that outputs compact occupancy queries which align with the language space, enabling end-to-end reasoning and forecasting via a Unified LLM. A novel LLM-guided Anchor-Diffusion Planner further fuses high-level semantic reasoning with diffusion-based trajectory refinement. Across scene understanding, occupancy forecasting, and open-loop planning, SparseOccVLA achieves state-of-the-art results while using far fewer visual tokens, highlighting the practicality of sparse occupancy-based cross-modal fusion. The approach offers a scalable, information-rich, and interpretable framework for holistic autonomous driving tasks.

Abstract

In autonomous driving, Vision Language Models (VLMs) excel at high-level reasoning , whereas semantic occupancy provides fine-grained details. Despite significant progress in individual fields, there is still no method that can effectively integrate both paradigms. Conventional VLMs struggle with token explosion and limited spatiotemporal reasoning, while semantic occupancy provides a unified, explicit spatial representation but is too dense to integrate efficiently with VLMs. To address these challenges and bridge the gap between VLMs and occupancy, we propose SparseOccVLA, a novel vision-language-action model that unifies scene understanding, occupancy forecasting, and trajectory planning powered by sparse occupancy queries. Starting with a lightweight Sparse Occupancy Encoder, SparseOccVLA generates compact yet highly informative sparse occupancy queries that serve as the single bridge between vision and language. These queries are aligned into the language space and reasoned by the LLM for unified scene understanding and future occupancy forecasting. Furthermore, we introduce an LLM-guided Anchor-Diffusion Planner featuring decoupled anchor scoring and denoising, as well as cross-model trajectory-condition fusion. SparseOccVLA achieves a 7% relative improvement in CIDEr over the state-of-the-art on OmniDrive-nuScenes, a 0.5 increase in mIoU score on Occ3D-nuScenes, and sets state-of-the-art open-loop planning metric on nuScenes benchmark, demonstrating its strong holistic capability.

Figures (9)

• Figure 1: The overview of SparseOccVLA, which is based on sparse occupancy queries, with a large language model serving as a unified token processor to simultaneously perform scene understanding, occupancy forecasting, and trajectory planning.
• Figure 2: Illustration of the Sparse Occupancy Encoder, where the distillation branch (indicated by gray) is removed during inference.
• Figure 3: The architecture of Diffusion Decoder.
• Figure 4: Ablation study on the number of queries (a), and the distillation loss (b).
• Figure 5: Qualitative results of SparseOccVLA for scene understanding and occupancy forecasting.