CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulation

TL;DR

CyberDemo presents a sim-to-real imitation learning pipeline that collects simulated human demonstrations, applies extensive trajectory-level data augmentation in a simulator, and fine-tunes on a small amount of real data to achieve robust dexterous manipulation. The approach uses automatic curriculum learning and Action Chunking with Transformers to produce smooth policies, and demonstrates superior real-world performance and generalization to unseen objects compared to real-data baselines. Key findings include a substantial in-domain performance boost, resilience to visual and pose variations, and effective handling of novel objects like tetra- and penta-valves. The work highlights the practical potential of simulator-based data augmentation for real-world dexterous manipulation and provides supplementary materials detailing data collection, augmentation derivations, and extensive ablations.

Abstract

We introduce CyberDemo, a novel approach to robotic imitation learning that leverages simulated human demonstrations for real-world tasks. By incorporating extensive data augmentation in a simulated environment, CyberDemo outperforms traditional in-domain real-world demonstrations when transferred to the real world, handling diverse physical and visual conditions. Regardless of its affordability and convenience in data collection, CyberDemo outperforms baseline methods in terms of success rates across various tasks and exhibits generalizability with previously unseen objects. For example, it can rotate novel tetra-valve and penta-valve, despite human demonstrations only involving tri-valves. Our research demonstrates the significant potential of simulated human demonstrations for real-world dexterous manipulation tasks. More details can be found at https://cyber-demo.github.io

Figures (13)

• Figure 1: We propose CyberDemo, a novel pipeline for learning real-world dexterous manipulation by using simulation data. First, we collect human demos in a simulated environment (blue region), followed by extensive data augmentation within the simulator (yellow region). Then, the imitation learning model, trained on augmented data and fine-tuned on a few real data, can be deployed on a real robot.
• Figure 2: CyberDemo Pipeline. First, we collect both simulated and real demonstrations via vision-based teleoperation. Following this, we train the policy on simulated data, incorporating the proposed data augmentation techniques. During training, we apply automatic curriculum learning, which incrementally enhances the randomness scale based on task performance. Finally, the policy is fine-tuned with a few real demos before being deployed to the real world.
• Figure 3: Data Augmentation. Our dataset augmentation encompasses four dimensions: (a) random camera views, (b) diverse objects, (c) random object pose, (d) random light and texture.
• Figure 4: Generalization to Novel Objects for Pick and Place. We compare our approach with the baselines in scenarios involving novel objects, random light disturbances, and random object positions.
• Figure 5: Generalization to Novel Objects for Rotating. The experimental setup for this task mirrors that of the "Generalization to Novel Objects for Pick and Place" experiments.