Precise-Physics Driven Text-to-3D Generation

TL;DR

Phy3DGen is proposed, a precise-physics-driven text-to-3D generation method that can consider both geometric plausibility and precise physics perception, further bridging 3D virtual modeling and precise physical worlds.

Abstract

Text-to-3D generation has shown great promise in generating novel 3D content based on given text prompts. However, existing generative methods mostly focus on geometric or visual plausibility while ignoring precise physics perception for the generated 3D shapes. This greatly hinders the practicality of generated 3D shapes in real-world applications. In this work, we propose Phy3DGen, a precise-physics-driven text-to-3D generation method. By analyzing the solid mechanics of generated 3D shapes, we reveal that the 3D shapes generated by existing text-to-3D generation methods are impractical for real-world applications as the generated 3D shapes do not conform to the laws of physics. To this end, we leverage 3D diffusion models to provide 3D shape priors and design a data-driven differentiable physics layer to optimize 3D shape priors with solid mechanics. This allows us to optimize geometry efficiently and learn precise physics information about 3D shapes at the same time. Experimental results demonstrate that our method can consider both geometric plausibility and precise physics perception, further bridging 3D virtual modeling and precise physical worlds.

Figures (7)

• Figure 1: Motivation. By applying distributed force on the object top, we analyze the solid mechanics of 3D shapes generated by text-to-3D generation methods by FEM. For physics, the lighter the color values, the higher the stress levels. We observe that the generated 3D shape by Diffusion-SDF li2023diffusion will experience notably higher stress in some regions, demonstrating that the generated geometry is fragile. In contrast, the stress distribution of our generated 3D shape is more uniform as the generated geometry conforms to the physical laws. Moreover, our learned physics information is comparable to the physics obtained by FEM.
• Figure 2: Overall framework of our method. Our method consists of two stages, initialization and optimization. We employ a 3D diffusion model to generate an initial mesh in the initialization stage. Then, the mesh is used to initialize the geometry network. Physics embedding is first computed to initialize the physics network in the optimization stage. Then, we sample 3D points to compute their geometry (SDF) and physics (displacement, strain, stress) properties. In this way, a total loss composed of geometry constraint loss, design loss, volume regularization loss, and physics loss is calculated to optimize the geometry and physics networks simultaneously.
• Figure 3: Qualitative geometry comparison.
• Figure 4: The physics predictions for the tear drop table (upper) and the two-layered table (lower) provided by the FEM (left), the physical layer without precise-physics embedding, and the physical layer with precise-physics embedding.
• Figure 5: Ablation study results.