From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation

TL;DR

The paper tackles the data efficiency bottleneck in dexterous manipulation by enabling imitation learning from human demonstrations via a vision-based, single-camera teleoperation system. It constructs a customized robot hand in a physical simulator that mirrors the operator's hand, allowing intuitive data collection that can later be retargeted to multiple manufactured hands offline. Demonstrations are used to train policies with Demo Augmented Policy Gradient (DAPG), improving sample efficiency and promoting human-like, safe behavior. The approach yields significant data collection gains (~60 demonstrations per hour), robust sim2real transfer to a real Allegro hand on a robotic arm, and strong performance across Relocate, Flip, and Open Door tasks, with demonstrated generalization to novel objects.

Abstract

We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This provides an intuitive interface and avoid unstable human-robot hand retargeting for data collection, leading to large-scale and high quality data. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured) to generate training demonstrations. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks. Importantly, we show our learned policy is significantly more robust when transferring to the real robot. More videos can be found in the https://yzqin.github.io/dex-teleop-imitation .

Figures (11)

• Figure 1: Overview: We introduce a teleoperation system which utilizes a single camera on an iPad to stream a human hand, estimates the hand pose and shape, and converts it to a customized robot hand in a physical simulator for dexterous manipulation. Once the manipulation trajectories are collected, we translate them to different specified robot hands to generate demonstrations, and use them to perform imitation learning on the same manipulation tasks. Once the policy is trained, we deploy it to the real robot hand and show robust transfer results.
• Figure 2: Overall Pipeline: We stream the hand of human operator with an RGB-D camera. First we construct a customized robot hand in a physical simulator from estimated hand shape parameters result and teleoperate this robot to perform dexterous manipulation task. After teleoperation, we translate the collected trajectory on the customized hand to three different robot hands using retargeting. Finally, we train individual policy on each hand using the translated demonstrations. The red and green curve in 2nd and 3rd rows represent the finger tip trajectory of thumb and pinkie. Different box color means different hand.
• Figure 3: Hardware setup with an iPad and a computer.
• Figure 4: Illustration of different customized robot hands generated from different human hands. The hand on left and right comes from different human. The red lines visualize the kinematics tree.
• Figure 5: Demonstration Collection and Translation: The top three rows shows camera stream, hand pose detection results, and the teleoperated customized robot hand in simulation. The bottom three rows shows the translated demonstration on three different robot hands by retargeting from the teleoperation trajectory.