UMat: Uncertainty-Aware Single Image High Resolution Material Capture

TL;DR

This work proposes a learning-based method to recover normals, specularity, and roughness from a single diffuse image of a material, using microgeometry appearance as the authors' primary cue, which shows outstanding performance integrating global information at reduced computational complexity.

Abstract

We propose a learning-based method to recover normals, specularity, and roughness from a single diffuse image of a material, using microgeometry appearance as our primary cue. Previous methods that work on single images tend to produce over-smooth outputs with artifacts, operate at limited resolution, or train one model per class with little room for generalization. Previous methods that work on single images tend to produce over-smooth outputs with artifacts, operate at limited resolution, or train one model per class with little room for generalization. In contrast, in this work, we propose a novel capture approach that leverages a generative network with attention and a U-Net discriminator, which shows outstanding performance integrating global information at reduced computational complexity. We showcase the performance of our method with a real dataset of digitized textile materials and show that a commodity flatbed scanner can produce the type of diffuse illumination required as input to our method. Additionally, because the problem might be illposed -- more than a single diffuse image might be needed to disambiguate the specular reflection -- or because the training dataset is not representative enough of the real distribution, we propose a novel framework to quantify the model's confidence about its prediction at test time. Our method is the first one to deal with the problem of modeling uncertainty in material digitization, increasing the trustworthiness of the process and enabling more intelligent strategies for dataset creation, as we demonstrate with an active learning experiment.

Figures (9)

• Figure 1: Our method digitizes a material taking as input a single scanned image. Further, it returns a pixel-wise metric of uncertainty ${\sigma_{\text{BRDF}}}$, computed at test time through probabilistic sampling, proven useful for active learning. In the plot we compare the average deviations of the radiance of different renders in the blue crop w.r.t the ground truth (GT) of: 1) the distribution of the probabilistic samples of a model trained with 100% of the data; 2) the deterministic output of that model; 3) the output of a model trained using 40% of the training dataset, sampled by active learning guided by ${\sigma_{\text{BRDF}}}$ and; 4) a model trained using 40% of the training dataset, randomly sampled. The material at the bottom, for which the model shows a higher uncertainty, generates more varied renders and differs most from the ground truth.
• Figure 2: Scanner images vs fitted albedos.
• Figure 3: Overview of UMat. We propose an attention-guided generator with a U-Net discriminator trained with style, frequency, pixel-wise, and adversarial losses. In green, we show the components that include any form of attention mechanism. The supplementary material contains the detailed architectures. On the right, we show two applications of our method: First, thanks to our test-time uncertainty evaluation, we can provide a measure of reliability of the estimation. Second, the maps that we produce can be used by any render engine.
• Figure 4: Top: input image of a rib material with metallic sequins. Bottom: ${\sigma_{\text{BRDF}}}$ and per-map uncertainties.
• Figure 5: Qualitative results of some configurations of our ablation study. In red, we show that the baseline generator architecture trained on the full loss introduces artifacts, which are removed using attention on the encoder. Further results are included in the supplementary material.