SoftMimicGen: A Data Generation System for Scalable Robot Learning in Deformable Object Manipulation

Abstract

Large-scale robot datasets have facilitated the learning of a wide range of robot manipulation skills, but these datasets remain difficult to collect and scale further, owing to the intractable amount of human time, effort, and cost required. Simulation and synthetic data generation have proven to be an effective alternative to fuel this need for data, especially with the advent of recent work showing that such synthetic datasets can dramatically reduce real-world data requirements and facilitate generalization to novel scenarios unseen in real-world demonstrations. However, this paradigm has been limited to rigid-body tasks, which are easy to simulate. Deformable object manipulation encompasses a large portion of real-world manipulation and remains a crucial gap to address towards increasing adoption of the synthetic simulation data paradigm. In this paper, we introduce SoftMimicGen, an automated data generation pipeline for deformable object manipulation tasks. We introduce a suite of high-fidelity simulation environments that encompasses a wide range of deformable objects (stuffed animal, rope, tissue, towel) and manipulation behaviors (high-precision threading, dynamic whipping, folding, pick-and-place), across four robot embodiments: a single-arm manipulator, bimanual arms, a humanoid, and a surgical robot. We apply SoftMimicGen to generate datasets across the task suite, train high-performing policies from the data, and systematically analyze the data generation system. Project website: \href{https://softmimicgen.github.io}{softmimicgen.github.io}.

Figures (3)

• Figure 1: SoftMimicGen System Pipeline.(Left) Human-teleoperated demonstrations are segmented into object-centric subtasks using manual annotations or heuristic signals, forming a library of source segments. (Right) Given a new target scene, SoftMimicGen(i) observes the current deformable state, (ii) performs non-rigid registration to establish correspondence and a warp field between the source and target geometries (correspondence + warp field), (iii) selects the source segment with the lowest registration cost and applies the resulting warp field to the end-effector trajectory (warp trajectory; source end-effector trajectory in light arrows, warped trajectory in darker arrows), and (iv) executes the warped trajectory in the target environment.
• Figure 2: Simulation Tasks.SoftMimicGen is used to generate new demonstrations across 10 challenging tasks involving 4 distinct robot embodiments: (a-b) GR1 humanoid, (c-f) Franka arm, (g-h) dVRK surgical robot, and (i-j) bimanual YAM arms. These tasks demand high precision and fine-grained manipulation to be successfully executed.
• Figure 3: Real-World Deployment. Example rollouts of real-world deformable manipulation tasks using policies trained on SoftMimicGen-generated datasets.