DemoBot: Efficient Learning of Bimanual Manipulation with Dexterous Hands From Third-Person Human Videos

TL;DR

DemoBot tackles the data bottleneck in dexterous manipulation by learning from a single unannotated RGB-D video. It converts the video into hand–object motion priors via a MANO-based hand retargeting and full-body trajectory generation, then refines these priors with a residual RL policy that adds a corrective action $\Delta a$ to the demonstration base $a_{demo}$. Three novel RL designs—Temporal-segment RL, Success-Gated Resets, and Event-Driven Reward curriculum—address long-horizon control, enabling efficient learning and precise manipulation across synchronous and asynchronous assembly tasks, with sim-to-real transfer verified on a UR-3e + XHand system. This approach demonstrates scalable skill acquisition from internet-scale visual data, bridging embodiment and dynamics gaps, and moving toward more general embodied AI.

Abstract

This work presents DemoBot, a learning framework that enables a dual-arm, multi-finger robotic system to acquire complex manipulation skills from a single unannotated RGB-D video demonstration. The method extracts structured motion trajectories of both hands and objects from raw video data. These trajectories serve as motion priors for a novel reinforcement learning (RL) pipeline that learns to refine them through contact-rich interactions, thereby eliminating the need to learn from scratch. To address the challenge of learning long-horizon manipulation skills, we introduce: (1) Temporal-segment based RL to enforce temporal alignment of the current state with demonstrations; (2) Success-Gated Reset strategy to balance the refinement of readily acquired skills and the exploration of subsequent task stages; and (3) Event-Driven Reward curriculum with adaptive thresholding to guide the RL learning of high-precision manipulation. The novel video processing and RL framework successfully achieved long-horizon synchronous and asynchronous bimanual assembly tasks, offering a scalable approach for direct skill acquisition from human videos.

Figures (9)

• Figure 1: The DemoBot framework for learning bimanual skills from a single visual demonstration. (a) The Data Processing Module converts a raw RGB-D video into structured motion priors in three steps: (1) A human demonstration is recorded and manually segmented with keyframes. (2) Hand and object estimators produce 3D hand and object poses, which are then refined using task-aware optimization. (3) The refined human motion is retargeted to the robot, generating a full-body trajectory that is split into meaningful temporal segments based on the keyframes. (b) The Corrective Residual RL Module then uses these segments as motion priors. The RL agent learns a corrective policy that outputs a residual action, $\Delta a$, which is added to the motion priors, $a = a_{demo} + \Delta a$, allowing the robot to master the contact-rich physical dynamics absent from the original visual data and complete the task.
• Figure 2: Robustness of the MANO-based hand pose estimation under different occlusion cases and hand gestures, with MANO hand model reconstructed from RGB images and visualized by overlaying on the original RGB images.
• Figure 3: Advantage of our algorithm preserving the integrity of the full hand pose: comparison of our developed MANO-based retargeting algorithms against the SOTA 2D keypoint-based retargeting.
• Figure 4: Physics-based simulation of bimanual assembly skills learned from human videos for the synchronous assembly task. The top row shows the collected demo and the rest rows show manipulation results with trained policy. We note that although our method is based on single demonstration, applying randomization on the initial poses of objects can still lead to generalization on random initial object poses.
• Figure 5: Physics-based simulation of bimanual assembly skills learned from human videos for the asynchronous assembly task. The top row shows the collected demo and the rest rows show manipulation results with trained policy. We note that although our method is based on single demonstration, applying randomization on the initial poses of objects can still lead to generalization on random initial object poses.