TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion

TL;DR

TextureDreamer tackles sparse-view texture transfer to arbitrary 3D geometry by combining Dreambooth-based texture extraction on 3–5 images with a geometry-aware diffusion-based synthesis stage. It represents textures as a neural BRDF field $f_{\theta}(v)$ mapping surface points $v$ to $(a,r,m)$ and optimizes via Personalized Geometry-aware Score Distillation (PGSD) conditioned on mesh normals through ControlNet. The core contributions are the geometry-aware diffusion framework, the integration of ControlNet-conditioned normal maps, and a streamlined training strategy that avoids heavy LoRA components for fidelity. This approach yields higher texture fidelity and 3D consistency across categories, enabling relightable textures for diverse assets and democratizing high-quality 3D texture generation for practical graphics pipelines.

Abstract

We present TextureDreamer, a novel image-guided texture synthesis method to transfer relightable textures from a small number of input images (3 to 5) to target 3D shapes across arbitrary categories. Texture creation is a pivotal challenge in vision and graphics. Industrial companies hire experienced artists to manually craft textures for 3D assets. Classical methods require densely sampled views and accurately aligned geometry, while learning-based methods are confined to category-specific shapes within the dataset. In contrast, TextureDreamer can transfer highly detailed, intricate textures from real-world environments to arbitrary objects with only a few casually captured images, potentially significantly democratizing texture creation. Our core idea, personalized geometry-aware score distillation (PGSD), draws inspiration from recent advancements in diffuse models, including personalized modeling for texture information extraction, variational score distillation for detailed appearance synthesis, and explicit geometry guidance with ControlNet. Our integration and several essential modifications substantially improve the texture quality. Experiments on real images spanning different categories show that TextureDreamer can successfully transfer highly realistic, semantic meaningful texture to arbitrary objects, surpassing the visual quality of previous state-of-the-art.

Figures (10)

• Figure 1: Texture transfer from sparse images. Given a small number of images and a target mesh, our method synthesizes geometry-aware texture that looks similar to the input appearances for diverse objects.
• Figure 2: Limitation of text-guided texturing. Compared to text-guided texturing method which requires a captioning method to generate a text prompt which might not express all the details of the image, image-based guided texturing can be more effective and more expressive. Image captioning is predicted by BLIP li2022blip, text-guided texturing is generated via TEXTure richardson2023texture, and image-guided result is from our method.
• Figure 3: Overview of TextureDreamer, a framework which synthesizes texture for a given mesh with appearance similar to 3-5 input images of an object. We first obtain personalized diffusion model $\psi$ with Dreambooth ruiz2023dreambooth finetuning on input images. The spatially-varying bidirectional reflectance distribution (BRDF) field $f_{\theta}$ for the 3D mesh $\mathcal{M}$ is then optimized through personalized geometric-aware score distillation (PGSD) (detailed in Section \ref{['sec:pgsd']}). After optimization finished, high-resolution texture maps corresponding to albedo, metallic, and roughness can be extracted from the optimized BRDF field.
• Figure 4: Image-guided transfer results from four categories (beds, sofas, plush toys, and mugs) of image sets to diverse objects. Our method can be applied to a wide range of object types and transfer the textures to diverse object shapes.
• Figure 5: Example of cross-category texture transfer results. In the first row, we transfer appearances from plush toys to cups and chairs. In the second row, special patterns from mugs are transferred to bears and chairs. In the thrid row, textures from input sofa are transferred to cups and bears.