QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer

TL;DR

The paper addresses transferring human dexterous manipulation to robot hands in simulation, a problem aggravated by highly constrained, discontinuous contact dynamics and morphology differences. It introduces a hierarchy of parameterized quasi-physical simulators and a physics curriculum that starts from relaxed relaxations and progressively tightens to realistic physics, with residual neural components to capture unmodeled effects. Through gradient-based optimization and MPC, the method achieves over 11% higher tracking success than the best prior baselines across Bullet and Isaac Gym, on tasks with non-trivial object motions and tool use. The approach also reveals that curriculum design and residual physics are critical, and ablation studies confirm each component's contribution. This work advances simulators as a collaborative partner in skill learning for embodied AI and offers a path toward more reliable sim-to-real transfer.

Abstract

We explore the dexterous manipulation transfer problem by designing simulators. The task wishes to transfer human manipulations to dexterous robot hand simulations and is inherently difficult due to its intricate, highly-constrained, and discontinuous dynamics and the need to control a dexterous hand with a DoF to accurately replicate human manipulations. Previous approaches that optimize in high-fidelity black-box simulators or a modified one with relaxed constraints only demonstrate limited capabilities or are restricted by insufficient simulation fidelity. We introduce parameterized quasi-physical simulators and a physics curriculum to overcome these limitations. The key ideas are 1) balancing between fidelity and optimizability of the simulation via a curriculum of parameterized simulators, and 2) solving the problem in each of the simulators from the curriculum, with properties ranging from high task optimizability to high fidelity. We successfully enable a dexterous hand to track complex and diverse manipulations in high-fidelity simulated environments, boosting the success rate by 11\%+ from the best-performed baseline. The project website is available at https://meowuu7.github.io/QuasiSim/.

Figures (22)

• Figure 1: By optimizing through a quasi-physical simulator curriculum, we successfully transfer human demonstrations to dexterous robot hand simulations. We enable accurate tracking of complex manipulations with changing contacts (Fig. (a)), non-trivial object motions (Fig. (b)) and intricate tool-using (Fig. (c,d)). Besides, our physics curriculum can substantially improve a failed baseline (Fig. (e,f)).
• Figure 2: The parameterized quasi-physical simulator relaxes the articulated multi rigid body dynamics as the parameterized point set dynamics, controls the contact behavior via an unconstrained parameterized spring-damper contact model, and compensates for unmodeled effects via parameterized residual physics networks. We tackle the difficult dexterous manipulation transfer problem via a physics curriculum.
• Figure 3: Point Set can flexibly adjust its states, avoid overfitting to data noise, and ease the difficulty brought by the morphology difference.
• Figure 4: Qualitative comparisons. Please refer to https://meowuu7.github.io/QuasiSim/ and https://youtu.be/Pho3KisCsu4 for animated results.
• Figure 5: (a) Qualitative comparisons between our full method and the ablated models; (b) Training loss curve comparisons; (c) Tracking loss curve comparisons.