MeshGen: Generating PBR Textured Mesh with Render-Enhanced Auto-Encoder and Generative Data Augmentation

TL;DR

MeshGen tackles the challenge of generating 3D meshes with consistent geometry and PBR textures from a single image. It introduces a render-enhanced point-to-shape auto-encoder with coarse-to-fine optimization and ray-based regularization, plus a geometry-aware image-to-shape diffusion model trained with geometric alignment and generative rendering augmentation. For texturing, it combines a reference-attention-guided multi-view generator with a multi-view PBR decomposer and UV-space inpainting to deliver relightable, seam-free textures. Experiments show state-of-the-art performance in both geometry and texture quality across public benchmarks, demonstrating practical potential for high-quality, PBR-textured meshes from limited data.

Abstract

In this paper, we introduce MeshGen, an advanced image-to-3D pipeline that generates high-quality 3D meshes with detailed geometry and physically based rendering (PBR) textures. Addressing the challenges faced by existing 3D native diffusion models, such as suboptimal auto-encoder performance, limited controllability, poor generalization, and inconsistent image-based PBR texturing, MeshGen employs several key innovations to overcome these limitations. We pioneer a render-enhanced point-to-shape auto-encoder that compresses meshes into a compact latent space by designing perceptual optimization with ray-based regularization. This ensures that the 3D shapes are accurately represented and reconstructed to preserve geometric details within the latent space. To address data scarcity and image-shape misalignment, we further propose geometric augmentation and generative rendering augmentation techniques, which enhance the model's controllability and generalization ability, allowing it to perform well even with limited public datasets. For the texture generation, MeshGen employs a reference attention-based multi-view ControlNet for consistent appearance synthesis. This is further complemented by our multi-view PBR decomposer that estimates PBR components and a UV inpainter that fills invisible areas, ensuring a seamless and consistent texture across the 3D mesh. Our extensive experiments demonstrate that MeshGen largely outperforms previous methods in both shape and texture generation, setting a new standard for the quality of 3D meshes generated with PBR textures. See our code at https://github.com/heheyas/MeshGen, project page https://heheyas.github.io/MeshGen

Figures (17)

• Figure 1: Overview of MeshGen. We first train a render-enhanced auto-encoder to compress meshes to more compact latent space (Section. \ref{['sec:autoencoder']}). We establish an image-to-shape diffusion model based on our tailored generative augmentations for improving controllability and generalization ability (Section. \ref{['sec:diffusion']}). The obtained mesh undergoes a reference attention-based multi-view synthesis and a PBR decomposer to obtain multi-view PBR channels. A UV-space inpainter is then exploited to fill the areas invisible in multi-view images (Section. \ref{['sec:texture']}).
• Figure 2: Illustration of the proposed ray-based regularization and two data augmentations.
• Figure 3: The effectiveness of the proposed reference attention fine-tuning. Here "ref attn" stands for reference attention, "f.t." denotes fine-tuning.
• Figure 4: Qualitative comparison on in-the-wild images with state-of-the-art large reconstruction models (upper part, including InstantMesh xu2024instantmesh, TripoSR TripoSR2024 and MeshFormer liu2024meshformer) and 3D native diffusion models (lower part, including CraftsMan li2024craftsman, LN3Diff lan2024ln3diff, and 3DTopia-XL chen2024primx). More comparisons and results are presented in the appendix \ref{['app:more_results']} and supplement video.
• Figure 5: Qualitative comparison with previous mesh texturing methods, including EASI-Tex perla2024easitex and Paint3D zeng2023paint3d.