EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects

TL;DR

EMPM presents a unified real-to-sim-to-real framework that combines an action-conditioned differentiable MPM simulator with geometry reconstruction and photorealistic appearance via Gaussian Splatting to model and simulate deformable objects. Material parameters, including $E$, $\nu$, $\rho$, and plastic yield things, are learned offline from multi-view RGB-D data and refined online from streaming observations, by minimizing a loss that aligns simulated deformations with observed geometry and appearance. The approach supports elastoplastic dynamics, large deformations, and contact through a differentiable pipeline, and demonstrates improved accuracy over spring-mass baselines in both offline and online settings, with practical robotic manipulation scenarios. This work enables physics-aware prediction and planning for complex deformables, facilitating robust robotic interaction and what-if analysis through a synchronized digital twin.

Abstract

Modeling deformable objects - especially continuum materials - in a way that is physically plausible, generalizable, and data-efficient remains challenging across 3D vision, graphics, and robotic manipulation. Many existing methods oversimplify the rich dynamics of deformable objects or require large training sets, which often limits generalization. We introduce embodied MPM (EMPM), a deformable object modeling and simulation framework built on a differentiable Material Point Method (MPM) simulator that captures the dynamics of challenging materials. From multi-view RGB-D videos, our approach reconstructs geometry and appearance, then uses an MPM physics engine to simulate object behavior by minimizing the mismatch between predicted and observed visual data. We further optimize MPM parameters online using sensory feedback, enabling adaptive, robust, and physics-aware object representations that open new possibilities for robotic manipulation of complex deformables. Experiments show that EMPM outperforms spring-mass baseline models. Project website: https://embodied-mpm.github.io.

Figures (6)

• Figure 1: Method Overview of EMPM. Our MPM simulation engine takes a reconstructed pointcloud and tracked 3D points as inputs. The model parameters are optimized using the discrepancy between the model's prediction and 3D shape reconstruction and tracking. The control inputs applied to the dynamics model of MPM is computed as the velocity extracted from the reconstructions. In the inference stage, we test the model's performance in terms of predicting 3D point positions and RGB image rendering of the 3D Gaussian splats.
• Figure 2: Objects used in our experiments. first row: elastic objects to simulate bending, poking and pulling; second row: elastoplastic objects to simulate fracture, stretch, and squeezing.
• Figure 3: Qualitative results showing the simulation predictions and real observations for different objects, comparing our method and PhysTwin jiang2025phystwin. Our method especially shines at modeling fracture, illustrated in the pita bread example, where PhysTwin model prediction completely fails. In the last frame of the plasticine example, the PhysTwin model prediction fails due to the over-bend of the internal spring model, where our model can still successfully model the squeezing of the object.
• Figure 4: Qualitative results demonstrate photorealistic reconstruction using Gaussian Splatting. Our reconstructed dynamics closely match real observations, whereas the spring-mass-based PhysTwinjiang2025phystwin fails to accurately model these materials.
• Figure 5: Qualitative results of the online simulation demonstrate the tracking performance of our MPM model against real observations. With online optimization guided by real-time reconstruction, the alignment between simulation and observation improves progressively over time. The losses are normalized from 0 to 1 for better illustration. Updates in the bread dough case are not obvious due to a relatively small motion and the occlusion of the grippers, it is recommended to refer to the video for a better dynamic visualization.