MVPainter: Accurate and Detailed 3D Texture Generation via Multi-View Diffusion with Geometric Control

TL;DR

MVPainter addresses the challenge of 3D texture generation by leveraging multi-view diffusion guided by explicit geometric conditioning and a dedicated PBR attribute extractor. It combines a robust data-processing pipeline with a Union ControlNet to fuse normal and depth priors across six views, trained in three stages, and builds high-quality texture data to improve detail and alignment. The approach achieves state-of-the-art performance in reference-texture alignment, geometry-texture consistency, and local texture quality, validated through VLM-based Elo evaluations and human studies, and extends to generating PBR maps for realistic rendering. The authors release an open-source system encompassing data construction, model architecture, and evaluation tools to promote reproducibility and further research.

Abstract

Recently, significant advances have been made in 3D object generation. Building upon the generated geometry, current pipelines typically employ image diffusion models to generate multi-view RGB images, followed by UV texture reconstruction through texture baking. While 3D geometry generation has improved significantly, supported by multiple open-source frameworks, 3D texture generation remains underexplored. In this work, we systematically investigate 3D texture generation through the lens of three core dimensions: reference-texture alignment, geometry-texture consistency, and local texture quality. To tackle these issues, we propose MVPainter, which employs data filtering and augmentation strategies to enhance texture fidelity and detail, and introduces ControlNet-based geometric conditioning to improve texture-geometry alignment. Furthermore, we extract physically-based rendering (PBR) attributes from the generated views to produce PBR meshes suitable for real-world rendering applications. MVPainter achieves state-of-the-art results across all three dimensions, as demonstrated by human-aligned evaluations. To facilitate further research and reproducibility, we also release our full pipeline as an open-source system, including data construction, model architecture, and evaluation tools.

Figures (15)

• Figure 1: MVPainter generates textures with excellent geometry alignment, reference fidelity, and rich detail, demonstrating strong capability in 3D texture synthesis.
• Figure 2: Three key challenges in existing 3D texture generation methods.
• Figure 3: Overview of our framework. First, we apply data filtering and augmentation strategies to ensure that the training data contains sufficient detail and variations in lighting and viewpoint. Then, we leverage a ControlNet-based architecture to generate multi-view texture images that are structurally consistent with the 3D geometry. Finally, we introduce a dedicated PBR extraction module to estimate the basecolor, metallic, and roughness maps, which are projected back onto the 3D mesh to obtain a PBR-textured model.
• Figure 4: Architecture of union controlnet. It performs multi-source information fusion through condition encoder and transformer, enabling support for diverse types of control signals as input.
• Figure 5: Example of the concatenated image which then is evaluated by VLM.