GenesisTex: Adapting Image Denoising Diffusion to Texture Space

TL;DR

GenesisTex adapts the pretrained image diffusion model to texture space by texture space sampling, which overcomes the limitations of slow optimization in distillation-based methods and instability in inpainting-based methods.

Abstract

We present GenesisTex, a novel method for synthesizing textures for 3D geometries from text descriptions. GenesisTex adapts the pretrained image diffusion model to texture space by texture space sampling. Specifically, we maintain a latent texture map for each viewpoint, which is updated with predicted noise on the rendering of the corresponding viewpoint. The sampled latent texture maps are then decoded into a final texture map. During the sampling process, we focus on both global and local consistency across multiple viewpoints: global consistency is achieved through the integration of style consistency mechanisms within the noise prediction network, and low-level consistency is achieved by dynamically aligning latent textures. Finally, we apply reference-based inpainting and img2img on denser views for texture refinement. Our approach overcomes the limitations of slow optimization in distillation-based methods and instability in inpainting-based methods. Experiments on meshes from various sources demonstrate that our method surpasses the baseline methods quantitatively and qualitatively.

Figures (19)

• Figure 1: Texturing results of different methods. Score Distillation Sampling (SDS) based method produces blurred and oversaturated textures. Inpainting-based approach results in artifacts at the boundaries of inpainting masks. Texture space sampling concurrently generating content from multiple viewpoints, produces clean, clear and natural colored textures.
• Figure 2: Overview of GenesisTex. GenesisTex generates a texture map for a given mesh $\mathcal{M}$, based on a prompt. Texture Space Sampling samples a texture map using Stable Diffusion, introducing style consistency and dynamic alignment across multiple viewpoints. Furthermore, Inpainting and Img2Img are applied to fill in the blank regions and enhance the quality of texture map details, respectively.
• Figure 3: Qualitative comparisons with Text2Mesh michel2022text2mesh, Latent-Paint metzer2023latent, Text2Tex chen2023text2tex and TEXTure richardson2023texture. In comparison with the baselines, our GenesisTex exhibits richer details and fewer artifacts.
• Figure 4: Texturing results with GenesisTex.
• Figure 5: Qualitative Comparisons with TexFusion cao2023texfusion. Images are extracted from their original paper.