MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training

TL;DR

MiVLA tackles the limited generalization of vision-language-action models for robotics by introducing human-robot mutual imitation pre-training. It bridges simulated robot data and real human behavior through bidirectional cross-embodiment action mapping and a diffusion-based action decoder, enabling the model to forecast trajectories for one embodiment and imitate another unseen embodiment. Experiments across RoboTwin-2.0 and three real robots demonstrate significant improvements over state-of-the-art VLAs in both simulation and real-world tasks, including effective few-shot adaptation. This approach shows a scalable path to generalizable VLAs without relying solely on real-world robot demonstrations, with potential for further integration with semantic guidance from vision-language models.

Abstract

While leveraging abundant human videos and simulated robot data poses a scalable solution to the scarcity of real-world robot data, the generalization capability of existing vision-language-action models (VLAs) remains limited by mismatches in camera views, visual appearance, and embodiment morphologies. To overcome this limitation, we propose MiVLA, a generalizable VLA empowered by human-robot mutual imitation pre-training, which leverages inherent behavioral similarity between human hands and robotic arms to build a foundation of strong behavioral priors for both human actions and robotic control. Specifically, our method utilizes kinematic rules with left/right hand coordinate systems for bidirectional alignment between human and robot action spaces. Given human or simulated robot demonstrations, MiVLA is trained to forecast behavior trajectories for one embodiment, and imitate behaviors for another one unseen in the demonstration. Based on this mutual imitation, it integrates the behavioral fidelity of real-world human data with the manipulative diversity of simulated robot data into a unified model, thereby enhancing the generalization capability for downstream tasks. Extensive experiments conducted on both simulation and real-world platforms with three robots (ARX, PiPer and LocoMan), demonstrate that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs (e.g., $\boldsymbolπ_{0}$, $\boldsymbolπ_{0.5}$ and H-RDT) by 25% in simulation, and 14% in real-world robot control tasks.

Figures (9)

• Figure 1: The motivation. (a) Vision-language-action models are generally driven by large-scale pre-training; however, their scaling is hindered by the scarcity of real-world robot demonstrations. (b) Simulated robot and human videos provide a promising alternative, as they offer not only robot action priors but also real-world behavior knowledge derived from human daily activities. (c) We propose human-robot mutual imitation to pre-train on simulation and human data, (d) achieving a generalizable model MiVLA with state-of-the-art manipulation performance on both simulation and real robot platforms.
• Figure 2: The overview of Proposed MiVLA. A general human-robot action mapping mechanism is introduced to bridge the gap between human-robot action space. Given a simulated robot demonstration, a VLA model is trained to predict robot action and learn to imitate robot behavior at human action space. For human demonstration, we train the same policy using human-to-robot imitation.
• Figure 3: The overview of three designed tasks across three different robots. Red arrows indicate robot actions.
• Figure 4: Qualitative investigation of generalization within three settings: cross-location generalization, cross-object generalization and cross-scene generalization.
• Figure A1: Comparison of the 'easy mode' (top-left) and 'hard mode' (remaining images) environments in RoboTwin-2.0.