Phy124: Fast Physics-Driven 4D Content Generation from a Single Image

TL;DR

Phy124 addresses the challenge of generating physically plausible 4D content from a single image by integrating a physics-based simulation into the generation pipeline and removing diffusion-model sampling during the 4D dynamics phase. It first constructs static 3D Gaussians guided by diffusion priors and then drives their motion with the Material Point Method (MPM) under externally applied forces, ensuring adherence to physical laws while enabling precise user control. Key contributions include a two-stage framework (3D Gaussians Generation followed by 4D Dynamics Generation), a plug-and-play architecture with multiple 3D backends, and a compact 4D representation for fast rendering, achieving about 39.5 seconds per instance and improved fidelity over state-of-the-art baselines. The approach holds practical significance for fast, controllable, physics-consistent 4D content creation in animation, gaming, and VR, without the heavy computational burden of diffusion-based dynamics.

Abstract

4D content generation focuses on creating dynamic 3D objects that change over time. Existing methods primarily rely on pre-trained video diffusion models, utilizing sampling processes or reference videos. However, these approaches face significant challenges. Firstly, the generated 4D content often fails to adhere to real-world physics since video diffusion models do not incorporate physical priors. Secondly, the extensive sampling process and the large number of parameters in diffusion models result in exceedingly time-consuming generation processes. To address these issues, we introduce Phy124, a novel, fast, and physics-driven method for controllable 4D content generation from a single image. Phy124 integrates physical simulation directly into the 4D generation process, ensuring that the resulting 4D content adheres to natural physical laws. Phy124 also eliminates the use of diffusion models during the 4D dynamics generation phase, significantly speeding up the process. Phy124 allows for the control of 4D dynamics, including movement speed and direction, by manipulating external forces. Extensive experiments demonstrate that Phy124 generates high-fidelity 4D content with significantly reduced inference times, achieving stateof-the-art performance. The code and generated 4D content are available at the provided link: https://anonymous.4open.science/r/BBF2/.

Figures (5)

• Figure 1: The figure shows a duck being pressed. Animate124 struggles to produce effective motion and often results in abnormal appearances in the 4D content; for instance, the duck in the figure has two beaks. On the other hand, DG4D produces motion that does not adhere to physical laws, such as the abnormal deformations observed in the duck's head and blue wing. In contrast, our approach can generate 4D content with greater physical accuracy. Moreover, our method reduces the generation time to an average of just 39.5 seconds.
• Figure 2: Framework of Phy124. In 3D Gaussians Generation stage, from an input image, a static 3D Gaussians will be generated under the guidance of the diffusion model. In 4D Dynamics Generation stage, we consider each 3D Gaussian kernel as particles within a continuum and attribute physical properties (e.g., density, mass, etc) to them. Sequentially, we employ MPM to introduce dynamics to the static 3D Gaussians. Meanwhile, users can guide the MPM simulator to generate 4D content that aligns with their desired outcomes by adjusting the external forces.
• Figure 3: Qualitative comparison with the baseline methods for image-to-4D generation. The description below the input image outlines the 4D content the user aims to generate. For each method, 14 frames of 4D content are generated, and every second frame is selected for display, showing a total of seven frames. Additionally, to compare geometric and temporal consistency across multiple views, the rendering perspective will change with each time step.
• Figure 4: 4D content generated by applying different external forces. $\mathbf{f}$ denotes the external forces applied along the $x$, $y$, and $z$ directions. Dashed lines are included in the images to help observe the motion.
• Figure 5: 4D content generated using different 3D generation methods.