Cross-Hand Latent Representation for Vision-Language-Action Models

TL;DR

XL-VLA is introduced, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands that is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data.

Abstract

Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.

Figures (16)

• Figure 1: Model Pipeline. XL-VLA builds on $\pi_0$pi0 with vision and language encoders paired with an action expert that operates in a shared latent action space for cross-embodiment control. During VLA training, the action expert is finetuned while the pretrained latent encoders and decoders remain frozen.
• Figure 2: Latent space pretraining pipeline. For each hand type, joint positions $\mathbf{q}_{h}$ are mapped through an encoder MLP into a shared latent space and reconstructed by a decoder MLP. The diagram also indicates the placement of the reconstruction loss $L_1$, retargeting loss $L_2$ via differentiable forward kinematics, and latent regularization loss $L_3$.
• Figure 3: Zero-shot Unseen Tasks Generalization. For each hand, we randomly select some tasks as unseen tasks, whose data are held out from the training dataset. Then we test the unseen tasks with model trained on other data. Results show that by training with an aligned latent action space, XL-VLA gets the ability to generalize to novel hand-task combination in a zero-shot manner. PSR stands for "Partial Success Rate", where policy is rewarded with half success if only one arm finishes its task.
• Figure 4: G1 Cross-Robot Performance. Co-training with latent xArm and humanoid data outperforms using raw actions.
• Figure 5: Latent Visualizations. Latent decoding results cross embodiment.