PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation

TL;DR

PhysFlow tackles the problem of physically realistic 4D dynamic scene simulation by marrying multi-modal foundation-model material inference with a differentiable MPM-based simulator, guided by optical-flow information within a video-diffusion loop. It initializes material types and properties via GPT-4, reconstructs scenes as 3D Gaussian splats, and iteratively refines parameters using a flow-based loss, avoiding memory-intensive render or SDS losses. The approach yields improved parameter identification, robust handling of large motions, and superior physical-realism and photo-realism on both synthetic and real-world data, outperforming strong baselines and demonstrating versatility across input types. This framework advances practical physics-aware rendering and robotics perception by enabling flexible, accurate 4D dynamics across diverse materials and scenarios.

Abstract

Realistic simulation of dynamic scenes requires accurately capturing diverse material properties and modeling complex object interactions grounded in physical principles. However, existing methods are constrained to basic material types with limited predictable parameters, making them insufficient to represent the complexity of real-world materials. We introduce PhysFlow, a novel approach that leverages multi-modal foundation models and video diffusion to achieve enhanced 4D dynamic scene simulation. Our method utilizes multi-modal models to identify material types and initialize material parameters through image queries, while simultaneously inferring 3D Gaussian splats for detailed scene representation. We further refine these material parameters using video diffusion with a differentiable Material Point Method (MPM) and optical flow guidance rather than render loss or Score Distillation Sampling (SDS) loss. This integrated framework enables accurate prediction and realistic simulation of dynamic interactions in real-world scenarios, advancing both accuracy and flexibility in physics-based simulations.

Figures (11)

• Figure 1: Overview of our proposed pipeline for 4D dynamic physical scene simulation. The process begins with 3D scene reconstruction using Gaussian splatting methods for different input types (multi-view images, dynamic video, and single image). Initial material properties are inferred through multi-modal foundation models and assigned to the reconstructed scene. The material parameters are optimized using optical flow guidance within a video diffusion framework, integrated with a differentiable MPM to ensure physically realistic simulation.
• Figure 2: Qualitative results of all methods on synthetic dataset.
• Figure 3: Comparisons of $\mathcal{L}_{render}$ and ours ($\mathcal{L}_{flow}$) with ECMS$\downarrow$.
• Figure 4: Ablations on physics reasoning, showing material values, timestep 30 frame, and deformation frequency.
• Figure 5: Qualitative results of all methods on real-world dataset. The yellow arrows show the input force for the simulated objects.