OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

TL;DR

OmniPhysGS introduces Constitutive Gaussians to extend 3D Gaussians with learnable constitutive models, enabling general physics-based 4D dynamics for heterogeneous materials. The framework integrates a physics-guided network (3D feature encoder + physical-aware decoder) with Material Point Method simulation and Score Distillation Sampling-guided diffusion supervision, allowing automatic, prompt-driven synthesis across elastic, viscoelastic, plastic, and fluid substances and their interactions. The approach demonstrates broad material generalization and improved visual-text alignment (3–16% gains) over state-of-the-art baselines, while maintaining physical plausibility through ensemble constitutive priors. This work enables more automatic, scalable, and physically faithful dynamic scene generation with potential applications in games, VR, robotics, and CAD tools.

Abstract

Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

Figures (48)

• Figure 1: Comparison with Previous Methods. Existing methods rely on handcrafted or narrowly restrictive physical models (e.g., pure elasticity) that limit generalizability. Our method introduces Constitutive Gaussians to better represent physical materials, thus achieving more automatic and physically plausible dynamic synthesis of various materials within a unified framework.
• Figure 2: Method Overview. OmniPhysGS extends 3D Gaussians with learnable constitutive models, introducing Constitutive Gaussians to the differentiable Material Point Method (MPM). A pre-trained video diffusion model is used to guide the training with Score Distillation Sampling.
• Figure 3: Constitutive Gaussian Network. The network architecture of Constitutive Gaussians consists of a 3D feature encoder and a physical-aware decoder. Expert-designed constitutive models are integrated into the decoder to guide the learning process, effectively avoiding the convergence issues faced by vanilla neural networks.
• Figure : “A swinging ficus.”
• Figure : “A lego excavator is digging soil.”