DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

TL;DR

DreamMat tackles the ill-posed problem of distilling PBR materials from diffusion-generated textures by conditioning a diffusion model on explicit geometry and a diverse set of environment lights, and by distilling with a Classifier Score Distillation (CSD) loss. It represents materials via a hash-grid SVBRDF and renders with Monte Carlo techniques, enforcing alignment with text prompts while disentangling shading from albedo. A geometry- and light-aware diffusion model (via ControlNet) ensures image outputs respect both surface geometry and lighting, enabling material parameters that render photorealistically under new illumination. Extensive experiments show improved material fidelity, visual quality, and editing capabilities, making DreamMat well-suited for game and film workflows, with some limitations in handling highly complex BRDFs, transparency, and indirect illumination.

Abstract

2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat, an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

Figures (20)

• Figure 1: Generated albedo and rendering results in the same environment light. (a) TEXTure yu2023texture generates an RGB texture map containing shading effects, leading to incorrect renderings in a new environment. (b) Fantasia3D Chen_2023_ICCV directly distills a diffusion model to generate materials, which still contain unwanted shading effects in albedo. (c) Our method can generate correct materials, allowing for more photorealistic renderings in a new environment.
• Figure 2: An untextured stool mesh and its generated images using different methods. (a) An untextured mesh with a given light environment. (b) A generated image of a depth-to-image Stable Diffusion model, which is inconsistent with the given environment light and results in incorrect materials decomposition. (c) An image generated by our geometry- and light-aware diffusion model, which is consistent with the environment light.
• Figure 3: Overview of our pipeline. DreamMat distills a diffusion model to generate PBR materials. We first use Monte Carlo sampling to render images of the object from its material representation and a randomly-selected predefined environment light. Then, we train the material representation by CSD loss on rendered images using a geometry- and light-aware diffusion model.
• Figure 4: Our geometry- and light-aware diffusion model uses an object's normal and depth maps as geometry conditions and six predefined materials with a given environment light as lighting conditions. Our model generates images that align with the given geometry and environment light.
• Figure 5: Qualitative comparison. We compared our method to TANGO chen2022tango, TEXTure yu2023texture, Text2Tex chen2023text2tex, and Fantasia3D Chen_2023_ICCV. We use NvDiffRec Munkberg_2022_CVPR to decompose the texture map produced by TEXTure and Text2Tex. Each object has three images: the albedo map on the left, the rendered image on the top right, and the roughness map on the bottom right.