Generalized Dynamics Generation towards Scannable Physical World Model

TL;DR

This work presents GDGen, a unified framework that casts diverse physical dynamics—soft, articulated, and rigid—as low-energy, potential-driven processes within a geometry-agnostic representation. It couples a neural deformation eigenmode field with a neural per-point material-field that predicts isotropic and directional stiffness, and optimizes an extended Neohookean energy $W_{\text{total}}$ that includes a directional stiffness term via $E_{aniso}$. By reconstructing observed dynamics and learning from motion, the method enables future-state prediction and generation of new dynamics under novel interactions, supported by a contrastive energy objective to discourage implausible dynamics. The approach advances scalable, generalizable simulation and robotics training in scannable digital twin environments, with potential benefits for realistic virtual worlds and embodied AI, while highlighting the need for collision handling and support for thin-shell geometries in future work.

Abstract

Digital twin worlds with realistic interactive dynamics presents a new opportunity to develop generalist embodied agents in scannable environments with complex physical behaviors. To this end, we present GDGen (Generalized Representation for Generalized Dynamics Generation), a framework that takes a potential energy perspective to seamlessly integrate rigid body, articulated body, and soft body dynamics into a unified, geometry-agnostic system. GDGen operates from the governing principle that the potential energy for any stable physical system should be low. This fresh perspective allows us to treat the world as one holistic entity and infer underlying physical properties from simple motion observations. We extend classic elastodynamics by introducing directional stiffness to capture a broad spectrum of physical behaviors, covering soft elastic, articulated, and rigid body systems. We propose a specialized network to model the extended material property and employ a neural field to represent deformation in a geometry-agnostic manner. Extensive experiments demonstrate that GDGen robustly unifies diverse simulation paradigms, offering a versatile foundation for creating interactive virtual environments and training robotic agents in complex, dynamically rich scenarios.

Figures (11)

• Figure 1: Compared to existing methods, our GDGen method enables different simulation capabilities.
• Figure 2: GDGen handles diverse dynamics and geometry representation.
• Figure 3: Overview of the GDGen algorithm: GDGen takes in a geometry $\mathbf{x}^{\text{rest}}$, and if trajectories are given, we reconstruct the observed motion $\{\mathbf{x}_t\}_t$ by optimizing both motion eigenmode weights $\mathbf{w}$ (output of the deformation net $f_{\theta_\mathbf{w}}$) and the transformation handles $\mathbf{T}$. We then use the predicted eigenmodes weights $\mathbf{w}$ and transformations $\mathbf{T}$ to calculate elasticity quantities, including deformation gradient $\mathbf{F}$, to be fed into the Young's modulus net $f_{\theta_\mathbf{E}}$ to predict the isotropic and anisotropic Young's Moduli $\mathbf{E}$. The predictions including motion eigenmode weights and Young's Moduli can be used to simulate new motion dynamics (right bottom). We mark the training objective of $f_{\theta_\mathbf{w}}$ in red color and objectives of $f_{\theta_\mathbf{E}}$ in blue color.
• Figure 4: Simulated reconstruction sequence of multi-body point clouds scene and articulated triangle mesh.
• Figure 5: Simulating articulated motion and predicting future dynamics with 3D gaussian splats.