Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting

TL;DR

<p>The paper addresses the challenge of simulating realistic physics-driven motion for 3D scenes without manual parameter tuning or heavy video priors. It introduces PhysSplat, a pipeline that first performs 3D open-vocabulary segmentation, then uses MLLM-based Physical Property Perception (MLLM-P3) to obtain mean material properties, followed by the Material Property Distribution Prediction (MPDP) to estimate a full distribution, and finally drives dynamics with Physical-Geometric Adaptive Sampling (PGAS) and MLS-MPM. Key contributions include the first zero-shot MLLM-based property estimation for 3D objects, distribution-based parameter modeling to reduce computation, and an adaptive sampling strategy for open-world, multi-object dynamics, achieving more realistic motion with significantly faster inference on a single GPU. The approach offers practical impact for immersive VR, robotics simulation, and AI-assisted scene understanding by enabling efficient, controllable, and physically plausible 4D reconstructions from static inputs.

Abstract

Recent advancements in 3D generation models have opened new possibilities for simulating dynamic 3D object movements and customizing behaviors, yet creating this content remains challenging. Current methods often require manual assignment of precise physical properties for simulations or rely on video generation models to predict them, which is computationally intensive. In this paper, we rethink the usage of multi-modal large language model (MLLM) in physics-based simulation, and present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics. We begin with detailed scene reconstruction and object-level 3D open-vocabulary segmentation, progressing to multi-view image in-painting. Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception (MLLM-P3) to predict mean physical properties of objects in a zero-shot manner. Based on the mean values and the object's geometry, the Material Property Distribution Prediction model (MPDP) model then estimates the full distribution, reformulating the problem as probability distribution estimation to reduce computational costs. Finally, we simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling (PGAS) strategy, efficiently capturing complex deformations and significantly reducing computational costs. Extensive experiments and user studies demonstrate our Sim Anything achieves more realistic motion than state-of-the-art methods within 2 minutes on a single GPU.

Figures (7)

• Figure 1: We develop an efficient method for simulating the dynamic movements of 3D objects with customizable behaviors, and synthesizing interactive 3D dynamics under arbitrary forces (red arrows). Compared to recent methods zhang2024physdreamerliu2024physics3dhuang2024dreamphysics, our approach produces more realistic 3D dynamics with much faster inference times.
• Figure 2: Overview of PhysSplat .Given a pre-trained 3D scene and its corresponding 2D images, we first perform object-level segmentation of the 3D scene with the prior from a set of foundation meodels liu2023groundingkirillov2023segmentzhang2024recognize. We obtain the mean physical properties of the object from the proposed MLLM-P3, and based on this and the object’s geometry, we then derive the full distribution using the MPDP model. Finally, we animate the 3D objects using a physics-based simulator with driving particles sampled via the Physical-Geometric Adaptive Sampling (PGAS) strategy.
• Figure 3: Sampling. We design a novel Physical-Geometric Adaptive Sampling (PGAS) strategy that captures the boundary of the object well. We employ PGAS to sample some “driving particles” (in green) and simulate only these particles. For rendering, each particle’s position and rotation are derived by fitting a local rigid body transformation based on neighboring driving particles.
• Figure 4: Qualitative Comparison on PhysDreamer zhang2024physdreamer. We compare our results with real captured videos, and some recent SOTA methods xie2024physgaussianzhang2024physdreamerliu2024physics3dren2023dreamgaussian4d. Our PhysSplat produces more realistic damping, closely matching real-world capture.
• Figure 5: Visual results on synthesized dataset liu2024physics3d with an external force (red arrows). PhysSplat is able to generate realistic scene movement while maintaining good motion consistency.