GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality

TL;DR

This work proposes a novel framework named GaussianDreamerPro, which binds Gaussians to reasonable geometry, which evolves over the whole generation process and shows significantly enhanced details and quality compared with previous methods.

Abstract

Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at https://taoranyi.com/gaussiandreamerpro/.

Figures (10)

• Figure 1: GaussianDreamerPro can generate high-quality 3D assets based on text and can be applied to downstream manipulation pipelines.
• Figure 2: We show the changes in 3D assets during the training process of GaussianDreamer yi2023gaussiandreamer and our method. Compared with GaussianDreamer, which grows Gaussians uncontrollably, resulting in always blurry edges, our method continuously improves the quality of appearance under the constraint of geometry.
• Figure 3: Our framework can be divided into two parts: basic 3D asset generation and quality enhancement with geometry-bound Gaussians. In the basic 3D asset generation stage, we generate initial 3D assets, which are used to initialize 2D Gaussians, obtain basic 3D assets under the optimization of the 2D diffusion model, and export as a mesh. In the quality enhancement with geometry-bound Gaussians stage, we bind 3D Gaussians to the mesh, and also obtain enhanced 3D assets under the optimization of the 2D diffusion model.
• Figure 4: Qualitative comparisons between our method and LucidDreamer EnVision2023luciddreamer, DreamCraft3D sun2023dreamcraft3d, DreamFusion poole2022dreamfusion, Magic3D lin2023magic3d, Fantasia3D chen2023fantasia3d , GaussianDreamer yi2023gaussiandreamer and GSGEN chen2023gsgen.
• Figure 5: Animation and simulation of the generated 3D assets.