SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation

TL;DR

SoMA introduces a real-to-sim neural simulator for soft-body manipulation that operates directly on Gaussian splats to jointly model deformable object dynamics, environmental forces, and robot actions. It uses a robot-conditioned scene-to-simulation mapping, a force-driven hierarchical GS dynamics model, and a multi-resolution training regime with occlusion-aware supervision and momentum regularization to achieve stable, long-horizon, action-conditioned resimulation. The approach yields state-of-the-art performance on real-world RGB and depth metrics, improves generalization to unseen manipulations by leveraging robot control signals, and enables challenging tasks like long-horizon T-shirt folding with stable geometry and dynamics. This framework offers practical benefits for simulation-driven robot learning and policy transfer in deformable object manipulation, advancing the realism and reliability of real-to-sim pipelines.

Abstract

Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.

Figures (4)

• Figure 1: SoMA is a GS neural simulator that reconstructs and simulates deformable object dynamics from real-world robot manipulation. Learning from multi-view RGB observations, it performs action-conditioned simulation directly on Gaussian splats, enabling interaction-consistent, stable long-horizon resimulation with higher-fidelity rendering under both seen and unseen manipulations.
• Figure 2: Framework of SoMA. SoMA takes RGB observations and robot joint-space actions collected from real-world manipulation as input (Left). It reconstructs deformable objects as hierarchical Gaussian splats, and propagates them through a neural simulator with supervision from rendering and dynamics (Middle). Object motion is driven by force-based interactions, where environmental and robot-induced forces act on splats to produce deformation (Right). A two-stage multi-resolution training strategy first captures global motion with large temporal gaps and then refines fine-grained dynamics under occlusion and contact using small gaps.
• Figure 3: Qualitative resimulation and generalization under robot manipulation. Left: resimulation on training trajectories. Right: generalization to unseen robot actions and contact configurations. Across diverse soft-body objects, including near-linear (rope), near-planar (cloth), and volumetric (doll) objects, SoMA produces stable, long-horizon simulations that closely match observed dynamics. PhysTwin shows deviations under complex or unseen interactions due to real-to-sim mismatch, while GausSim often remains static or unstable in challenging scenarios.
• Figure 4: More Reults (a) multi-view results; (b) T-shirt folding comparison results (c) results on phystwin datatests.