Few-Shot Neural Differentiable Simulator: Real-to-Sim Rigid-Contact Modeling

TL;DR

The proposed few-shot real-to-sim approach combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)-based models to suggest that differentiable simulation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.

Abstract

Accurate physics simulation is essential for robotic learning and control, yet analytical simulators often fail to capture complex contact dynamics, while learning-based simulators typically require large amounts of costly real-world data. To bridge this gap, we propose a few-shot real-to-sim approach that combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)-based models. Using only a small amount of real-world data, our method calibrates analytical simulators to generate large-scale synthetic datasets that capture diverse contact interactions. On this foundation, we introduce a mesh-based GNN that implicitly models rigid-body forward dynamics and derive surrogate gradients for collision detection, achieving full differentiability. Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories. In addition, the differentiable design supports gradient-based optimization, which we validate through simulation-based policy learning in multi-object interaction scenarios. Extensive experiments show that our framework not only improves simulation fidelity with minimal supervision but also increases the efficiency of policy learning. Taken together, these findings suggest that differentiable simulation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.

Figures (7)

• Figure 1: Top: A blue cube is pushed to collide with a green cube, with the objective of stopping it at the red target area. The initial pushing velocity of the blue cube is optimized using stochastic gradient descent in our differentiable simulator, successfully achieving the goal. Bottom: Convergence of optimization across multiple runs, showing the loss curve (left) and the magnitude of the initial pushing velocity (right).
• Figure 2: Illustration of our framework. (1) We utilize sampling-based identification of contact parameters to minimize the trajectory loss between real world and simulation, followed by (2) data scaling in terms of quantity, geometry, $etc.$ (3) The GNN is trained on the scaled dataset, equipped with differential collision detection and shape matching to achieve differentiability. The resulting simulator is capable of accurately modeling complex contact interactions.
• Figure 3: (a) We collect real-world data on a tabletop setup with two cubes, considering a quasi-planar frictional contact scenario where one cube is pushed towards another cube at rest. (b) The poses of the cubes are estimated using TagSLAM.
• Figure 4: Comparison of trajectory errors (see Eq. \ref{['eq:objective']}) in MuJoCo before and after contact parameter identification. The identified parameters significantly enhance the simulation's accuracy in replicating real-world contact dynamics.
• Figure 5: Illustration of a scenario where the moving cube (green) collides with the static cube (red) before and after contact parameter identification.