$Δ$VLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation

TL;DR

A prior-guided framework that models world-knowledge variations relative to an explicit current-world knowledge prior for action generation, rather than regressing absolute future world states is proposed, and the Prior-Guided WorldKnowledge Extractor is proposed.

Abstract

Recent vision-language-action (VLA) models have significantly advanced robotic manipulation by unifying perception, reasoning, and control. To achieve such integration, recent studies adopt a predictive paradigm that models future visual states or world knowledge to guide action generation. However, these models emphasize forecasting outcomes rather than reasoning about the underlying process of change, which is essential for determining how to act. To address this, we propose $Δ$VLA, a prior-guided framework that models world-knowledge variations relative to an explicit current-world knowledge prior for action generation, rather than regressing absolute future world states. Specifically, 1) to construct the current world knowledge prior, we propose the Prior-Guided WorldKnowledge Extractor (PWKE). It extracts manipulable regions, spatial relations, and semantic cues from the visual input, guided by auxiliary heads and prior pseudo labels, thus reducing redundancy. 2) Building upon this, to represent how world knowledge evolves under actions, we introduce the Latent World Variation Quantization (LWVQ). It learns a discrete latent space via a VQ-VAE objective to encode world knowledge variations, shifting prediction from full modalities to compact latent. 3)Moreover, to mitigate interference during variation modeling, we design the Conditional Variation Attention (CV-Atten), whichpromotes disentangled learning and preserves the independence of knowledge representations. Extensive experiments on both simulated benchmarks and real-world robotic tasks demonstrate $Δ$VLA achieves state-of-the-art performance while improving efficiency. Code and real-world execution videos are available at https://github.com/JiuTian-VL/DeltaVLA.

Figures (8)

• Figure 1: Comparison with Previous Paradigms. (a) Image-based paradigms predict subgoal images for action generation. (b) World knowledge-based paradigms replace pixel-level prediction with future world states. (c) $\boldsymbol{\Delta}$VLA constructs an explicit current-world knowledge prior via PWKE and models discrete world-knowledge variations via LWVQ, enabling prior-grounded reasoning about how the world should change under actions. (d) Comparison of performance and efficiency with previous methods.
• Figure 2: Overview of $\Delta$VLA Framework. It first extracts current world knowledge via the Prior-Guided World Knowledge Extractor, which encodes semantic, regional, and depth cues from SigLIP and DINOv2 guided by auxiliary heads and prior pseudo labels to form a unified world prior. These representations are concatenated with world variation tokens and modeled through the Conditional Variation Attention for structured perception–variation reasoning. Finally, the Latent World Variation Quantization module learns a discrete codebook representing world knowledge variations, providing causal guidance for variation token learning and consistent action generation.
• Figure 3: Illustration of PWKE. World tokens have different meanings across encoders: they correspond to semantic tokens in SigLIP and depth tokens in DINOv2. The dashed box indicates auxiliary heads used only for training supervision.
• Figure 4: Illustration of CV-Atten. In red dashed boxes, each variation token attends to its corresponding world prior, ensuring type-specific modeling and minimizing cross-modality leakage. For action generation, we employ parallel decoding.
• Figure 5: Visualization of Manipulation Region Extraction and World Variation Modeling. On the left, the manipulable regions extracted by the PWKE module are shown. On the right, the future states, reconstructed from the current state and latent variation through the LWVQ module, closely align with the real future states and accurately highlight dynamic changes, marked by red circles.