RoboMirror: Understand Before You Imitate for Video to Humanoid Locomotion

TL;DR

RoboMirror tackles the gap between video understanding and humanoid control by replacing brittle pose-based retargeting with a principled, end-to-end video-to-locomotion pipeline. A Vision-Language Model extracts semantic latents from videos, which are then reconstructed into motion latents via a diffusion model, providing robust, semantically grounded conditioning for a diffusion-based policy. The approach combines a MoE-based teacher with a diffusion-based student trained through a DAgger-like paradigm, enabling retargeting-free imitation and telepresence from both egocentric and third-person videos. Extensive experiments show dramatic latency reduction, improved task success, and strong sim-to-real transfer, including real-world deployment on a Unitree G1. By anchoring control in visual understanding, RoboMirror closes the loop between perception and action for humanoid locomotion.

Abstract

Humans learn locomotion through visual observation, interpreting visual content first before imitating actions. However, state-of-the-art humanoid locomotion systems rely on either curated motion capture trajectories or sparse text commands, leaving a critical gap between visual understanding and control. Text-to-motion methods suffer from semantic sparsity and staged pipeline errors, while video-based approaches only perform mechanical pose mimicry without genuine visual understanding. We propose RoboMirror, the first retargeting-free video-to-locomotion framework embodying "understand before you imitate". Leveraging VLMs, it distills raw egocentric/third-person videos into visual motion intents, which directly condition a diffusion-based policy to generate physically plausible, semantically aligned locomotion without explicit pose reconstruction or retargeting. Extensive experiments validate the effectiveness of RoboMirror, it enables telepresence via egocentric videos, drastically reduces third-person control latency by 80%, and achieves a 3.7% higher task success rate than baselines. By reframing humanoid control around video understanding, we bridge the visual understanding and action gap.

Figures (10)

• Figure 1: RoboMirror makes humanoid understand before imitating. It acts like a mirror, which can not only infer and replicate the actions being performed by the shooter from egocentric videos based on the changes in the surrounding environmental perspective (as shown in the upper part of the figure), but also understand the actions first and then imitate them from third-person videos (as shown in the lower part of the figure), without the need for pose estimation and retargeting during inference.
• Figure 2: Overview of RoboMirror. It adopts a two-stage framework: initially, it leverages Qwen3-VL to process egocentric or third-person video inputs, generating motion latents through diffusion models with DiT $\mathcal{D}_\theta$. Subsequently, in the policy learning stage, a MoE-based teacher policy is trained with RL, while a diffusion-based student policy learns to denoise actions under the guidance of reconstructed motion latents. During inference, it can first understand and then imitate the motion in the video without motion obtainment and retargeting.
• Figure 3: Qualitative results in the IsaacGym and MuJoCo. The upper half presents the tracking performance of egocentric videos-to-locomotion, and the lower half presents that of third-person videos-to-locomotion.
• Figure 4: Qualitative results of generated motions.
• Figure 5: Qualitative results of PHC and GMR retargeting.