Contact-Aware Neural Dynamics

TL;DR

This work tackles the challenging sim-to-real gap in contact-rich robotic manipulation by introducing a contact-aware neural dynamics model that fuses simulation-derived priors with real tactile data. The model operates in two stages: a Contact Predictor that forecasts future contact events and a diffusion-based Pose Predictor that generates long-horizon object motions conditioned on those contacts and a multimodal history. Training proceeds first in large-scale simulation with domain randomization, followed by fine-tuning on limited real-world data to align simulated and real contact dynamics, yielding improved forward predictions and policy transfer. The approach demonstrates strong improvements in both single- and multi-object scenarios, exhibiting better realism, stability, and applicability to policy evaluation and refinement in real-world manipulation tasks.

Abstract

High-fidelity physics simulation is essential for scalable robotic learning, but the sim-to-real gap persists, especially for tasks involving complex, dynamic, and discontinuous interactions like physical contacts. Explicit system identification, which tunes explicit simulator parameters, is often insufficient to align the intricate, high-dimensional, and state-dependent dynamics of the real world. To overcome this, we propose an implicit sim-to-real alignment framework that learns to directly align the simulator's dynamics with contact information. Our method treats the off-the-shelf simulator as a base prior and learns a contact-aware neural dynamics model to refine simulated states using real-world observations. We show that using tactile contact information from robotic hands can effectively model the non-smooth discontinuities inherent in contact-rich tasks, resulting in a neural dynamics model grounded by real-world data. We demonstrate that this learned forward dynamics model improves state prediction accuracy and can be effectively used to predict policy performance and refine policies trained purely in standard simulators, offering a scalable, data-driven approach to sim-to-real alignment.

Figures (4)

• Figure 1: Overview of the proposed contact-aware neural dynamics framework. The model takes as input a multimodal history $\mathcal{H}_t=\{\mathbf{s}_{t-K:t},\,\mathbf{q}_{t-K:t},\,\mathbf{a}_{t-K:t},\,c_{t-K:t},\,\mathcal{P}\}$, including past object poses, joint values, robot actions, binary contact signals, and the object point cloud. A temporal encoder extracts features from the state--action--contact sequence, while a PointNet encoder processes the geometry $\mathcal{P}$. Their fused latent representation $\mathbf{z}_t$ is used by a contact prediction module to infer future contacts $\hat{c}_{t+1:t+H}$, which then condition a diffusion-based pose predictor that outputs future pose increments $\Delta\hat{\mathbf{s}}_{t+1:t+H}$. The model is first trained on large-scale simulation data and subsequently fine-tuned with a small amount of real-world interaction data, enabling implicit alignment of simulated and physical contact dynamics.
• Figure 2: Real-world setup with an XArm7 robot arm and an XHand equipped with tactile sensors for contact detection. A Realsense camera captures visual observations, and everyday YCB objects are used for grasping.
• Figure 3: Qualitative comparison between real, simulated, and our contact-aware neural dynamics results. The first row shows real-world rollouts, the second shows standard MuJoCo simulations, and the third shows our model predictions. Standard simulation often yields unstable or incorrect contacts and physically implausible object motion due to limited contact modeling and the sim-to-real gap. In contrast, our model produces smoother, more realistic trajectories aligned with real-world motion. When co-trained with a small amount of real data, it further improves temporal stability and contact consistency, demonstrating stronger sim-to-real transfer.
• Figure 4: Comparison of multi-trajectory rollouts: the left panel shows predictions from our two-stage contact-aware model versus ground truth, while the right panel shows rollouts from a direct dynamics predictor compared with the same ground truth.