DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning

TL;DR

DexMimicGen addresses the data bottleneck in training policies for bimanual dexterous robots by transforming a handful of human demonstrations into thousands of simulation trajectories. It introduces asynchronous per-arm execution, coordination synchronization, and sequential ordering to synthesize realistic multi-arm trajectories from limited data. Nine simulation environments across three embodiments demonstrate versatility, generating 21K demos from 60 source demonstrations and enabling a real2sim2real pipeline that achieves high success in a can-sorting task. The results, including comparisons to baselines and real-world tests with a digital twin, highlight the practical viability and potential for advancing data-efficient dexterous manipulation research.

Abstract

Imitation learning from human demonstrations is an effective means to teach robots manipulation skills. But data acquisition is a major bottleneck in applying this paradigm more broadly, due to the amount of cost and human effort involved. There has been significant interest in imitation learning for bimanual dexterous robots, like humanoids. Unfortunately, data collection is even more challenging here due to the challenges of simultaneously controlling multiple arms and multi-fingered hands. Automated data generation in simulation is a compelling, scalable alternative to fuel this need for data. To this end, we introduce DexMimicGen, a large-scale automated data generation system that synthesizes trajectories from a handful of human demonstrations for humanoid robots with dexterous hands. We present a collection of simulation environments in the setting of bimanual dexterous manipulation, spanning a range of manipulation behaviors and different requirements for coordination among the two arms. We generate 21K demos across these tasks from just 60 source human demos and study the effect of several data generation and policy learning decisions on agent performance. Finally, we present a real-to-sim-to-real pipeline and deploy it on a real-world humanoid can sorting task. Generated datasets, simulation environments and additional results are at https://dexmimicgen.github.io/

Figures (7)

• Figure 1: DexMimicGen Overview. DexMimicGen offers an efficient pipeline to train capable bimanual dexterous robots. (left) First, a human operator collects around five task demonstrations using a teleoperation device. (middle) Next, DexMimicGen automatically generates a large set of demonstration trajectories in simulation. (right) Finally, a policy is trained with imitation learning and deployed in the real world.
• Figure 2: Subtask Types. We categorize the subtasks into parallel, coordination, and sequential subtasks, where the two arms achieve subgoals independently, with coordination, and following a specific order.
• Figure 3: DexMimicGen Workflow. Left: segment source demonstrations for each arm through manually defined heuristics or human and records the poses of the reference objects. Right: In a new simulation environment, we generate trajectories by transforming source trajectories with reference object poses and executing them.
• Figure 4: Simulation Tasks. We deploy DexMimicGen on nine simulation tasks across three embodiments --- two arms with parallel-jaw grippers (top), two arms with dexterous hands (middle), and a humanoid (bottom)
• Figure 5: Dataset Size Comparison. Success rates of policies trained on datasets with different sizes.