X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real

TL;DR

X-Sim tackles learning robot manipulation from action-less human videos by leveraging object motion as a dense supervisory signal to train in a photorealistic simulation, then distilling into an image-conditioned policy for real-world deployment. The framework comprises real-to-sim transfer, privileged-state RL to generate synthetic image-action data, and sim-to-real transfer with online domain adaptation to align observations. Across five tasks in two environments, X-Sim yields average task-progress gains over hand-tracking baselines, matches behavior-cloning performance with an order of magnitude less data, and generalizes to novel viewpoints and test-time changes. This approach reduces reliance on robot teleoperation and offers a scalable path to adaptation for foundation-model–driven robotics in real-world settings.

Abstract

Human videos offer a scalable way to train robot manipulation policies, but lack the action labels needed by standard imitation learning algorithms. Existing cross-embodiment approaches try to map human motion to robot actions, but often fail when the embodiments differ significantly. We propose X-Sim, a real-to-sim-to-real framework that uses object motion as a dense and transferable signal for learning robot policies. X-Sim starts by reconstructing a photorealistic simulation from an RGBD human video and tracking object trajectories to define object-centric rewards. These rewards are used to train a reinforcement learning (RL) policy in simulation. The learned policy is then distilled into an image-conditioned diffusion policy using synthetic rollouts rendered with varied viewpoints and lighting. To transfer to the real world, X-Sim introduces an online domain adaptation technique that aligns real and simulated observations during deployment. Importantly, X-Sim does not require any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2 environments and show that it: (1) improves task progress by 30% on average over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with 10x less data collection time, and (3) generalizes to new camera viewpoints and test-time changes. Code and videos are available at https://portal-cornell.github.io/X-Sim/.

Figures (12)

• Figure 1: Overview of X-Sim: We introduce X-Sim, a real-to-sim-to-real framework that bridges the human-robot embodiment gap by learning robot policies. Real-to-Sim. We generate photorealistic simulation using object-centric rewards generated from human videos. Training X-Sim. We first train RL policies with privileged state using GPU-parallelized environment. Then, we collect a diverse image-action dataset use it to distill behaviors into an image-conditioned policy. Sim-to-Real. Image-based policy is deployed in the real-world. Its observation encoder automatically calibrates itself by aligning real and sim image observations at test-time.
• Figure 2: Real-to-Sim:X-Sim reconstructs a photorealistic environment with accurate geometry from multi-view images. It tracks object motion across time from an RGBD human video to define a dense object-centric reward function to train RL policies in simulation.
• Figure 3: Sim-to-Real:(Left)X-Sim distills privileged-state policies into image-conditioned policies by generating and a synthetic image-action dataset using multiple environment randomizations. (Right) During deployment, real policy rollouts are replayed in simulation to generate paired images across real and sim. Their discrepancy is utilized to minimize and calibrate the sim-to-real visual gap.
• Figure 4: Performance on Real-World Tasks: We report Avg. Task Progress on 5 tasks across two environments, and find that X-Sim both with and without calibration consistently outperforms hand-tracking baselines that attempt to retarget human hand motion. A rough sketch of each task is visualized on top.
• Figure 5: Hand Re-targeting Failure Modes:Hand Mask fails due to a significant visual domain gap between human and robots, even when the motions are physically feasible for the robot. Object-Aware IK fails under execution mismatch, as certain human hand motions are kinematically or dynamically infeasible.