Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimization

TL;DR

The paper tackles data scarcity in contact-rich robotic manipulation by proposing a low-cost, physics-driven data generation pipeline that starts from a handful of embodiment-agnostic VR demonstrations. It combines kinematic retargeting to target embodiments with demonstration-guided local trajectory optimization to produce thousands of dynamically feasible trajectories across varying physical parameters, enabling cross-embodiment data transfer. The approach is validated by training diffusion-based behavior cloning policies on the generated data, achieving strong simulation performance and zero-shot hardware success on bimanual iiwa arms, and showing significant improvements over baselines trained only on original demonstrations. This work reduces hardware data collection costs, supports legacy data reuse, and advances cross-embodiment generalization for contact-rich manipulation tasks.

Abstract

We present a low-cost data generation pipeline that integrates physics-based simulation, human demonstrations, and model-based planning to efficiently generate large-scale, high-quality datasets for contact-rich robotic manipulation tasks. Starting with a small number of embodiment-flexible human demonstrations collected in a virtual reality simulation environment, the pipeline refines these demonstrations using optimization-based kinematic retargeting and trajectory optimization to adapt them across various robot embodiments and physical parameters. This process yields a diverse, physically consistent dataset that enables cross-embodiment data transfer, and offers the potential to reuse legacy datasets collected under different hardware configurations or physical parameters. We validate the pipeline's effectiveness by training diffusion policies from the generated datasets for challenging contact-rich manipulation tasks across multiple robot embodiments, including a floating Allegro hand and bimanual robot arms. The trained policies are deployed zero-shot on hardware for bimanual iiwa arms, achieving high success rates with minimal human input. Project website: https://lujieyang.github.io/physicsgen/.

Figures (10)

• Figure 1: Physics-driven data generation overview. Leveraging trajectory optimization, our framework automatically generates thousands of dynamically feasible contact-rich trajectories across a range of embodiments and physical parameters from only 24 human demonstrations. The policy trained with imitation learning from the generated dataset is more robust and performant.
• Figure 2: VR-based human-hand demonstration framework.
• Figure 3: Human hand demo in VR and kinematic retargeting for different embodiments. The blue spheres illustrate the demo hand landmarks scaled to the specific system.
• Figure 4: Trajectory optimization is crucial for generating dynamically feasible trajectories. (Top) Before trajectory optimization, the kinematically retargeted demos easily lose contact and drive the object out of reach with different physical parameters or slight deviations in object states. (Bottom) Trajectory optimization encourages robots to establish contact with and maintain good manipulability of the object. The tricolor axis indicates the object orientation.
• Figure 5: Distribution and snapshots of trajectories generated from a single demonstration. (a) The original demonstration (orange) is locally perturbed and augmented to about 100 dynamically feasible contact-rich trajectories (blue) for each system. The density map represents the object pose distribution of the generated trajectories in the specific 2-dimensional slices. (b) Snapshots of 30 dynamically feasible trajectories under random physical parameters and object initial poses for bimanual iiwa arms are visualized.