SuperMat: Physically Consistent PBR Material Estimation at Interactive Rates

TL;DR

<3-5 sentence high-level summary> SuperMat addresses the challenge of fast, physically consistent PBR material estimation from 2D images and extends to 3D objects. It uses a single-step diffusion-based framework with structural expert branches to predict albedo, metallic, and roughness maps, trained end-to-end with perceptual and re-render losses; it also corrects the DDIM scheduler to enable one-step inference. For 3D assets, it introduces SuperMatMV and a UV refinement network that backprojects multi-view predictions into UV space and completes missing regions in a single step, achieving approximately 3 seconds per object. Across synthetic and real datasets, it achieves state-of-the-art decomposition quality with millisecond per-image speeds, enabling practical material acquisition for 3D assets.

Abstract

Decomposing physically-based materials from images into their constituent properties remains challenging, particularly when maintaining both computational efficiency and physical consistency. While recent diffusion-based approaches have shown promise, they face substantial computational overhead due to multiple denoising steps and separate models for different material properties. We present SuperMat, a single-step framework that achieves high-quality material decomposition with one-step inference. This enables end-to-end training with perceptual and re-render losses while decomposing albedo, metallic, and roughness maps at millisecond-scale speeds. We further extend our framework to 3D objects through a UV refinement network, enabling consistent material estimation across viewpoints while maintaining efficiency. Experiments demonstrate that SuperMat achieves state-of-the-art PBR material decomposition quality while reducing inference time from seconds to milliseconds per image, and completes PBR material estimation for 3D objects in approximately 3 seconds. The project page is at https://hyj542682306.github.io/SuperMat/.

Figures (17)

• Figure 1: We present an efficient method for high-quality material decomposition in both 2D and 3D. Left: Albedo and RM(Roughness, Metallic) predictions of SuperMat on rendered images under unknown illumination. The first row shows inputs with uneven lighting, the second row displays albedo predictions, and the third row presents RM predictions. Each texel in the RM map here is represented as $(1, \text{roughness}, \text{metallic})$. Middle: Rendering results of a 3D object with PBR materials obtained by our method under novel lighting. Right: Our method is efficient and produces high-quality results.
• Figure 2: Overview of the SuperMat framework. The UNet architecture incorporates structural expert branches for albedo and RM estimation, enabling parallel material property prediction while sharing a common backbone. During training, we leverage a fixed DDIM scheduler and optimize the network end-to-end using both perceptual loss and re-render loss. The HTML]f7e1edmarked region shows the inference pipeline, where our model achieves single-step inference for concurrent material map generation. Input geometric information (normal, position) and environment maps are used to compute the re-render loss.
• Figure 3: Material Decomposition for 3D objects. Applying SuperMat's powerful decomposition to 3D objects, we can obtain an object with high-quality materials in just three inference steps. Due to implementation differences, the texels in the RM map for image space decomposition are $(1, \text{roughness}, \text{metallic})$, while in the UV refinement process, they are $(0, \text{roughness}, \text{metallic})$.
• Figure 4: Comparison of our method with others on albedo estimation results. Each pair of rows presents results of the same object under identical lighting conditions from different viewpoints. This organizational format is also followed in other qualitative comparison figures.
• Figure 5: Comparison of our method with others on metallic and roughness estimation results. We combined both materials into a single image, with the left half representing metallic and the right half showing roughness. It can be observed that these two materials exhibit more variation and are more challenging to estimate compared to albedo.