You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations

TL;DR

YOTO enables rapid learning of complex bimanual manipulation from a single human video demonstration by extracting hand motions, converting them into discrete keyframes, and coordinating dual arms with a motion mask. It significantly expands training data through auto-rollout and geometry-based object transforms, feeding a diffusion-based BiDP that uses object-centric point clouds to predict sequences of keyposes. The approach demonstrates strong performance and generalization across five long-horizon tasks, outperforming state-of-the-art visuomotor imitation baselines. This framework offers a scalable, one-shot pathway to versatile dual-arm manipulation in real-world settings.

Abstract

Bimanual robotic manipulation is a long-standing challenge of embodied intelligence due to its characteristics of dual-arm spatial-temporal coordination and high-dimensional action spaces. Previous studies rely on pre-defined action taxonomies or direct teleoperation to alleviate or circumvent these issues, often making them lack simplicity, versatility and scalability. Differently, we believe that the most effective and efficient way for teaching bimanual manipulation is learning from human demonstrated videos, where rich features such as spatial-temporal positions, dynamic postures, interaction states and dexterous transitions are available almost for free. In this work, we propose the YOTO (You Only Teach Once), which can extract and then inject patterns of bimanual actions from as few as a single binocular observation of hand movements, and teach dual robot arms various complex tasks. Furthermore, based on keyframes-based motion trajectories, we devise a subtle solution for rapidly generating training demonstrations with diverse variations of manipulated objects and their locations. These data can then be used to learn a customized bimanual diffusion policy (BiDP) across diverse scenes. In experiments, YOTO achieves impressive performance in mimicking 5 intricate long-horizon bimanual tasks, possesses strong generalization under different visual and spatial conditions, and outperforms existing visuomotor imitation learning methods in accuracy and efficiency. Our project link is https://hnuzhy.github.io/projects/YOTO.

Figures (14)

• Figure 1: The overview of our proposed YOTO. It is a general framework consists of three main modules: (a) the human hand motion extraction and injection, (b) the training demonstration proliferation from one-shot teaching, and (c) the training and deployment of a customized bimanual diffusion policy (BiDP). It is best to zoom in to view the details.
• Figure 2: A detailed example of extracted motion trajectories with corresponding keyframes of both left hand and right hand. It is best to zoom in to view the details.
• Figure 3: We collected a variety of manipulated objects in instance-level for each of five bimanual tasks to improve and verify the generalizability of trained policies. All of these objects are from everyday life, not intentionally customized.
• Figure 4: Illustrations of extracted hand motion trajectories by using (a) unhandled raw 3D hand center points, (b) projected hand center points on the 2D image, and (c) lifted 3D points in simplified keyframes. The first and second line represents the task pull drawer and uncover lid, respectively.
• Figure 5: Ablation studies on expanded training data at different scales using geometric transformations. The task pull drawer with 243 episodes is treated as the not expanded version.