TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting

TL;DR

TexGaussian addresses the challenge of generating high-fidelity PBR materials directly on untextured 3D meshes. It introduces octant-aligned 3D Gaussian Splatting and a regression-trained octree-based 3D U-Net to predict per-octant Gaussian parameters encoding albedo, roughness, and metallic, followed by fast texture baking into UV space. The method achieves superior visual quality and significantly faster inference compared with diffusion-based and point-cloud approaches, and supports text-conditioned PBR material generation. This approach enables efficient, geometry-consistent material synthesis suitable for modern rendering pipelines and large-scale 3D content creation.

Abstract

Physically Based Rendering (PBR) materials play a crucial role in modern graphics, enabling photorealistic rendering across diverse environment maps. Developing an effective and efficient algorithm that is capable of automatically generating high-quality PBR materials rather than RGB texture for 3D meshes can significantly streamline the 3D content creation. Most existing methods leverage pre-trained 2D diffusion models for multi-view image synthesis, which often leads to severe inconsistency between the generated textures and input 3D meshes. This paper presents TexGaussian, a novel method that uses octant-aligned 3D Gaussian Splatting for rapid PBR material generation. Specifically, we place each 3D Gaussian on the finest leaf node of the octree built from the input 3D mesh to render the multi-view images not only for the albedo map but also for roughness and metallic. Moreover, our model is trained in a regression manner instead of diffusion denoising, capable of generating the PBR material for a 3D mesh in a single feed-forward process. Extensive experiments on publicly available benchmarks demonstrate that our method synthesizes more visually pleasing PBR materials and runs faster than previous methods in both unconditional and text-conditional scenarios, exhibiting better consistency with the given geometry. Our code and trained models are available at https://3d-aigc.github.io/TexGaussian.

Figures (12)

• Figure 1: Our proposed TexGaussian is capable of generating high-quality materials for given input 3D meshes based on the corresponding textual descriptions. The generated results are naturally compatible with modern graphical engines for photo-realistic rendering under different environment maps.
• Figure 2: An overview of our PBR material generation framework. (a) We propose octant-aligned 3D Gaussian Splatting, which positions a 3D Gaussian at the center of each finest leaf node of the constructed octree. Additional channels are added at the end of the Gaussian parameters to model PBR material. (b) We use the 3D U-Net built upon octree-based convolutional networks to predict the Gaussian parameters. Our octree-based 3D U-Net is trained by minimizing the difference on 2D raster images and 3D Gaussian parameters. (c) We bake the multi-view rendered images to the UV space of the input 3D model to realize physically based rendering under new illumination environments.
• Figure 3: Unconditional RGB texture generative results on ShapeNet. Please zoom in for a better inspection of color details.
• Figure 4: Examples of generated RGB textures obtained by TexGaussian and other state-of-the-art models on the same 3D object. Please zoom in for a better inspection.
• Figure 5: Qualitative comparison with Fantasia3D chen2023fantasia3d, FlashTex deng2024flashtex, DreamMat zhang2024dreammat and Make-it-Real fang2024makeitreal. We show the rendered image, albedo map, roughness map, and metallic map for each 3D object.