Visual Imitation Enables Contextual Humanoid Control

TL;DR

VideoMimic introduces a real-to-sim-to-real pipeline that converts monocular videos into environment-conditioned humanoid control policies. It jointly reconstructs metric 4D human motion and surrounding geometry, retargets motions to a humanoid, and trains a unified policy that uses a local height-map and root-direction to execute context-aware skills such as stair climbing and chair sitting. The resulting policy transfers to a real Unitree G1, demonstrating robust, context-aware behaviors in unseen environments without explicit task labels. The work also analyzes limitations of monocular reconstruction, retargeting, and simulation fidelity, and outlines directions for richer perception and multi-modal extensions.

Abstract

How can we teach humanoids to climb staircases and sit on chairs using the surrounding environment context? Arguably, the simplest way is to just show them-casually capture a human motion video and feed it to humanoids. We introduce VIDEOMIMIC, a real-to-sim-to-real pipeline that mines everyday videos, jointly reconstructs the humans and the environment, and produces whole-body control policies for humanoid robots that perform the corresponding skills. We demonstrate the results of our pipeline on real humanoid robots, showing robust, repeatable contextual control such as staircase ascents and descents, sitting and standing from chairs and benches, as well as other dynamic whole-body skills-all from a single policy, conditioned on the environment and global root commands. VIDEOMIMIC offers a scalable path towards teaching humanoids to operate in diverse real-world environments.

Figures (9)

• Figure 1: VideoMimic is a real-to-sim-to-real pipeline that converts monocular videos into transferable humanoid skills, letting robots learn context-aware behaviors (terrain-traversing, stairs-climbing, sitting) in a single policy. Video results are available on our webpage: https://videomimic.net.
• Figure 2: VideoMimic Real-to-Sim. A casually captured phone video provides the only input. We first reconstruct per-frame human motion and 2D keypoints, along with a dense scene point cloud. An efficient optimization jointly aligns the motion and point cloud, recovers statistically accurate metric scale using a human height prior, and registers the human trajectory based on human-associated points. The point cloud is then converted to a mesh, aligned with gravity, and the motion is retargeted to a humanoid in the reconstructed scene. This yields world-frame trajectories and simulator-ready meshes that serve as inputs for policy training.
• Figure 3: Policy training in sim. Our pipeline of training RL starts with a dataset of Motion Capture trajectories. We then inject a heightmap observation and track whole-body reference trajectories from our videos in various environments. We proceed to distill a policy conditioned only on the root position of the robot. We then finetune this policy directly with RL using the same reduced observation set. Our pipeline is motivated by three goals: (a) producing motions that are fast and faithful to the original video demonstrations; (b) ensuring observations are available in real-world settings; and (c) training a generalist policy that distills knowledge from all video demonstrations into a single model applicable beyond the training set.
• Figure 4: Versatile capabilities of our Real-to-Sim pipeline. VideoMimic enables (i) robust tracking of Internet videos with challenging motion and diverse environments, (ii) simultaneous reconstruction and retargeting of multiple humans, and (iii) ego-view RGB-D rendering for embodied perception---though not used in our current policy, it highlights the framework’s broader applicability across inputs and tasks.
• Figure 5: The policy performing various skills on the real robot: traversing complex terrain, standing, and sitting. All these skills are in a single policy, which decides what to do based on the context of its heightmap and joystick direction input. Top row: the policy stands from a seated position after sitting down. Second row: the policy walks up a flight of stairs. Third row: the policy walks down a flight of stairs. Bottom row: the policy walks over a kerb and onto a rough terrain. Please find the video results on our https://www.videomimic.net/.