villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

TL;DR

Villa-X presents a Vision-Language-Latent-Action framework that grounds latent actions in both visual changes and robot proprioception via a proprioceptive forward dynamics model and embodiment-aware context. It then couples a latent-action expert with a robot-action expert through a joint diffusion policy, enabling robust, zero-shot generalization to unseen embodiments and symbolic concepts. Across SIMPLER simulations and real-world robots, villa-X achieves state-of-the-art performance and demonstrates strong transfer from unlabeled video to robot control. The work advances ViLLA by tightly integrating physical grounding into latent representations and mid-level planning, paving the way for scalable, generalizable robot manipulation.

Abstract

Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.

Figures (13)

• Figure 1: (a) A standard Latent Action Model (LAM) learns a latent action $z_t$ primarily through visual reconstruction, predicting a future frame $\hat{o}_{t+K}$ from the current frame $o_t$ and latent action $z_t$. (b) Our proposed model enhances this by adding a proprio-FDM. This auxiliary module predicts future robot states $\hat{q}_{t+1:t+K}$ and actions $\hat{a}_{t:t+K-1}$ conditioned on an embodiment context $c_e$, enabling the latent actions to be better grounded in physical dynamics.
• Figure 2: Architecture of ACT: A hierarchical policy that predicts latent action plans and conditions robot action generation on them, incorporating embodiment context and attention masking.
• Figure 3: Probing experiment results.
• Figure 4: Visualization of zero-shot latent plans on an unseen embodiment. Each pair of images shows the starting frame (left) and the ending frame (right), with the instruction displayed above.
• Figure 5: Real-world robot evaluation platforms: (top) Realman robot arm platform with a gripper and (bottom) Xarm robot arm with Xhand dexterous hand. Platform setups are shown on the left, with corresponding evaluation tasks on the right.