RoboWheel: A Data Engine from Real-World Human Demonstrations for Cross-Embodiment Robotic Learning

TL;DR

RoboWheel tackles the bottleneck of scalable, cross-embodiment robotic supervision by turning real-world HOI videos into physically plausible, retargetable trajectories. It fuses monocular HOI reconstruction with TSDF-based optimization and a residual RL policy, then maps trajectories to diverse robots and enhances data through simulation-driven augmentation. The authors introduce HORA, a large multimodal HOI-to-robot dataset, and demonstrate that HOI-derived supervision can match teleoperation in stability and improve robustness on VLA/IL tasks. Overall, RoboWheel provides a scalable, embodiment-agnostic data engine that bridges real-world manipulation signals and cross-robot learning, enabling broader generalization across tasks and morphologies.

Abstract

We introduce Robowheel, a data engine that converts human hand object interaction (HOI) videos into training-ready supervision for cross morphology robotic learning. From monocular RGB or RGB-D inputs, we perform high precision HOI reconstruction and enforce physical plausibility via a reinforcement learning (RL) optimizer that refines hand object relative poses under contact and penetration constraints. The reconstructed, contact rich trajectories are then retargeted to cross-embodiments, robot arms with simple end effectors, dexterous hands, and humanoids, yielding executable actions and rollouts. To scale coverage, we build a simulation-augmented framework on Isaac Sim with diverse domain randomization (embodiments, trajectories, object retrieval, background textures, hand motion mirroring), which enriches the distributions of trajectories and observations while preserving spatial relationships and physical plausibility. The entire data pipeline forms an end to end pipeline from video,reconstruction,retargeting,augmentation data acquisition. We validate the data on mainstream vision language action (VLA) and imitation learning architectures, demonstrating that trajectories produced by our pipeline are as stable as those from teleoperation and yield comparable continual performance gains. To our knowledge, this provides the first quantitative evidence that HOI modalities can serve as effective supervision for robotic learning. Compared with teleoperation, Robowheel is lightweight, a single monocular RGB(D) camera is sufficient to extract a universal, embodiment agnostic motion representation that could be flexibly retargeted across embodiments. We further assemble a large scale multimodal dataset combining multi-camera captures, monocular videos, and public HOI corpora for training and evaluating embodied models.

Figures (20)

• Figure 1: Given monocular RGB(-D) input, we first estimate the motion of the hand or wholebody and the manipulated object. We then perform a joint optimization, guided by TSDF and reinforcement learning, to improve physical plausibility and ensure robotic reachability. The resulting trajectories are retargeted to heterogeneous embodiments—including arms, dexterous hands, and humanoids. Finally, domain randomization for both observation and trajectories in Isaac Sim is applied to enrich observational diversity for robotic arms, and the generated embodied data is validated across both VLA and IL policy benchmarks.
• Figure 2: Map the hand’s joints to the gripper’s end-effector pose, including the corresponding mapping of the gripper’s opening and closing state.
• Figure 3: RoboWheel diverse augmentation in simulation.
• Figure 4: Data collection setup and tactile information.
• Figure 5: HOI reconstruction results of RoboWheel .Whether the data comes from public HOI datasets (e.g., DexYCB) or not, RoboWheel can achieve high-precision HOI reconstruction.