Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process

TL;DR

The paper tackles unified vision-language-action modeling by introducing UD-VLA, which blends understanding, future-state image generation, and action prediction into a single joint discrete diffusion process (JD3P). By encoding all modalities into a shared discrete token space and employing a hybrid attention mechanism, the model enables tight cross-modal synergy where actions are refined under continual guidance from predicted visuals. A two-stage training pipeline and several inference-time techniques balance performance and efficiency, yielding state-of-the-art results on CALVIN, LIBERO, and SimplerEnv with substantially faster inference than autoregressive baselines. The approach demonstrates strong long-horizon planning and real-world generalization, illustrating significant practical potential for embodied AI systems.

Abstract

Vision-language-action (VLA) models aim to understand natural language instructions and visual observations and to execute corresponding actions as an embodied agent. Recent work integrates future images into the understanding-acting loop, yielding unified VLAs that jointly understand, generate, and act -- reading text and images and producing future images and actions. However, these models either rely on external experts for modality unification or treat image generation and action prediction as separate processes, limiting the benefits of direct synergy between these tasks. Our core philosophy is to optimize generation and action jointly through a synchronous denoising process, where the iterative refinement enables actions to evolve from initialization, under constant and sufficient visual guidance. We ground this philosophy in our proposed Unified Diffusion VLA and Joint Discrete Denoising Diffusion Process (JD3P), which is a joint diffusion process that integrates multiple modalities into a single denoising trajectory to serve as the key mechanism enabling understanding, generation, and acting to be intrinsically synergistic. Our model and theory are built on a unified tokenized space of all modalities and a hybrid attention mechanism. We further propose a two-stage training pipeline and several inference-time techniques that optimize performance and efficiency. Our approach achieves state-of-the-art performance on benchmarks such as CALVIN, LIBERO, and SimplerEnv with 4$\times$ faster inference than autoregressive methods, and we demonstrate its effectiveness through in-depth analysis and real-world evaluations. Our project page is available at https://irpn-eai.github.io/UD-VLA.github.io/.

Figures (5)

• Figure 1: Overview of our Unified Diffusion VLA. We first construct a unified multimodal space by quantizing multimodal information into discrete tokens. We then formalize a Joint Discrete Denoising Diffusion Process (JD3P) to allow visual generation and action prediction to be intrinsically synergistic. Moreover, we build our model on a pre-trained VLM and conduct two-stage training. Finally, we balance performance and efficiency during inference through several key techniques.
• Figure 2: Hybrid attention mechanism in UD-VLA.
• Figure 3: Real-world Setup and Results.
• Figure 4: Visualization of Generated Future Frames and Ground-truth Frames. The leftmost frame of each trajectory represents the current observation.
• Figure 5: Qualitative Comparison of discrete diffusion and continuous diffusion.