H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos

TL;DR

H2R addresses the visual domain gap between human egocentric hands and robotic arms by converting human hand videos into robot-centric frames through HaMeR-based pose estimation, inpainting, and precise overlay of simulated robot arms. The authors construct H2R-1M datasets across multiple robot embodiments and egocentric sources (SSv2, Ego4D) and introduce a CLIP-based semantic metric to assess fidelity. With MAE and R3M pretraining, models trained on H2R-enhanced data show consistent improvements in both simulation and real-world manipulation tasks, validating improved generalization and embodied transfer. This work reduces reliance on robot-specific demonstrations and demonstrates a scalable path for robot pretraining using large-scale human video datasets.

Abstract

Large-scale pre-training using videos has proven effective for robot learning. However, the models pre-trained on such data can be suboptimal for robot learning due to the significant visual gap between human hands and those of different robots. To remedy this, we propose H2R, a simple data augmentation technique that detects human hand keypoints, synthesizes robot motions in simulation, and composites rendered robots into egocentric videos. This process explicitly bridges the visual gap between human and robot embodiments during pre-training. We apply H2R to augment large-scale egocentric human video datasets such as Ego4D and SSv2, replacing human hands with simulated robotic arms to generate robot-centric training data. Based on this, we construct and release a family of 1M-scale datasets covering multiple robot embodiments (UR5 with gripper/Leaphand, Franka) and data sources (SSv2, Ego4D). To verify the effectiveness of the augmentation pipeline, we introduce a CLIP-based image-text similarity metric that quantitatively evaluates the semantic fidelity of robot-rendered frames to the original human actions. We validate H2R across three simulation benchmarks: Robomimic, RLBench and PushT and real-world manipulation tasks with a UR5 robot equipped with Gripper and Leaphand end-effectors. H2R consistently improves downstream success rates, yielding gains of 5.0%-10.2% in simulation and 6.7%-23.3% in real-world tasks across various visual encoders and policy learning methods. These results indicate that H2R improves the generalization ability of robotic policies by mitigating the visual discrepancies between human and robot domains.

Figures (9)

• Figure 1: Overview of H2R.H2R is a data augmentation technique designed to enhance robot pre-training by converting first-person human hand operation videos into robot-centric visual data. By bridging the visual domain gap, H2R improves pre-trained visual encoders for downstream robot policies (imitation learning), validated across simulation benchmarks and real-world robotic tasks.
• Figure 2: H2R Pipeline. H2R involves replacing human hands with robotic arms by first using the HaMeR model to detect hand poses and camera parameters. The human hand is then removed using the SAM, and the inpainting model LaMa fills in the gap. A robot hand is constructed based on the detected pose and keypoints, with the camera perspective adjusted to match the original image. Finally, the robot hand is overlaid onto the image, ensuring accurate alignment with the human hand.
• Figure 3: Examples of H2R Augmentation. Each column shows images before and after augmentation. The top row uses UR5 Leaphand, the middle row uses UR5 Gripper, and the bottom row uses Franka to replace human hands.
• Figure 4: Simulation Benchmark Overview. Visualization of the seven simulation tasks used for evaluation, including three from Robomimic (MoveCan, Square, Lift), one PushT task from Diffusion Policy, and three from RLBench (Close Box, Close Laptop Lid, Toilet Seat Down).
• Figure 5: Visualization of Real-world Manipulation Tasks. The left columns show Gripper tasks and the right columns show Leaphand tasks. Each task is illustrated with six frames, demonstrating the progression from the initial state to the completion of the manipulation.