DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation

TL;DR

DreamMesh is a novel text-to-3D architecture that pivots on well-defined surfaces (triangle meshes) to generate high-fidelity explicit 3D model and significantly outperforms state-of-the-art text-to-3D methods in faithfully generating 3D content with richer textual details and enhanced geometry.

Abstract

Learning radiance fields (NeRF) with powerful 2D diffusion models has garnered popularity for text-to-3D generation. Nevertheless, the implicit 3D representations of NeRF lack explicit modeling of meshes and textures over surfaces, and such surface-undefined way may suffer from the issues, e.g., noisy surfaces with ambiguous texture details or cross-view inconsistency. To alleviate this, we present DreamMesh, a novel text-to-3D architecture that pivots on well-defined surfaces (triangle meshes) to generate high-fidelity explicit 3D model. Technically, DreamMesh capitalizes on a distinctive coarse-to-fine scheme. In the coarse stage, the mesh is first deformed by text-guided Jacobians and then DreamMesh textures the mesh with an interlaced use of 2D diffusion models in a tuning free manner from multiple viewpoints. In the fine stage, DreamMesh jointly manipulates the mesh and refines the texture map, leading to high-quality triangle meshes with high-fidelity textured materials. Extensive experiments demonstrate that DreamMesh significantly outperforms state-of-the-art text-to-3D methods in faithfully generating 3D content with richer textual details and enhanced geometry. Our project page is available at https://dreammesh.github.io.

Figures (10)

• Figure 1: Existing methods poole2022dreamfusionmetzer2022latent-nerfwang2022SJClin2022magic3dfantasia3dwang2023prolificdreamer mostly hinge on implicit or hybrid 3D representation and produce noisy surfaces. Instead, our DreamMesh pivots on completely explicit 3D representation, yielding high-quality 3D meshes that exhibit clean, organized topology, devoid of any redundant vertices & faces.
• Figure 2: An overview of our DreamMesh that fully capitalizes on explicit 3D scene representation (triangle meshes) for text-to-3D generation in a coarse-to-fine scheme. In the first coarse stage, DreamMesh learns text-guided Jacobians matrices to deform a base mesh into the coarse mesh, and then textures it through a tuning-free process. In the second fine stage, both coarse mesh and texture are jointly optimized, yielding high-quality mesh with high-fidelity texture.
• Figure 3: Qualitative comparison of texture and wireframe results (rendering in Blender) between our DreamMesh and other baseline methods.
• Figure 4: Ablation study of our DreamMesh given the same text prompt. DreamMesh$_{coarse}^{-}$ is a degraded version of coarse stage that jointly optimizes mesh deformation and textures via SDS. DreamMesh$_{coarse}$ is the complete coarse stage that decouples the learning of coarse meshes and textures. DreamMesh is our full run with both coarse and fine stages.
• Figure 5: Comparisons between our DreamMesh and the integration of the state-of-the-art text driven mesh deformation technique TextDeformer and the advanced texturing methods of TEXTure TEXTure or Text2Tex chen2023text2tex for text-to-3D generation.