Neural Appearance Modeling From Single Images

Abstract

We propose a material appearance modeling neural network for visualizing plausible, spatially-varying materials under diverse view and lighting conditions, utilizing only a single photograph of a material under co-located light and view as input for appearance estimation. Our neural architecture is composed of two network stages: a network that infers learned per-pixel neural parameters of a material from a single input photograph, and a network that renders the material utilizing these neural parameters, similar to a BRDF. We train our model on a set of 312,165 synthetic spatially-varying exemplars. Since our method infers learned neural parameters rather than analytical BRDF parameters, our method is capable of encoding anisotropic and global illumination (inter-pixel interaction) information into individual pixel parameters. We demonstrate our model's performance compared to prior work and demonstrate the feasibility of the render network as a BRDF by implementing it into the Mitsuba3 rendering engine. Finally, we briefly discuss the capability of neural parameters to encode global illumination information.

Figures (9)

• Figure 1: The first stage of our architecture, $M_{\text{est}}$, which transforms an input RGB image and map of half-direction cosines into a map of implicit neural material parameters, to be rendered by $M_{\text{render}}$. The yellow layers represent Highlight-Aware residual Convolution blocks of 2 HA convolutions each. The blue layers represent standard residual convolution blocks.
• Figure 2: MLP Per-pixel Neural Renderer (NBRDF). The neural parameters (purple) are concatenated with the MLP-compressed encoding of light and view directions (orange); this input is fed into a 6-layer MLP to return a log-relative linear-RGB color (far right).
• Figure 3: LPIPS metric comparison of our model, Zhou and Kalantari's Adversarial Model zhou2021adversarial, Matfusion sartor2023matfusion, and Neural Relighting bieron2023single. Columns correspond to reflection sampling, identity, and random hemisphere sampling.
• Figure 4: Qualitative comparison of global illumination transport and shadows, Non-GI render (left), Blender (middle) against our model (right). Note the shadow effect of lower orange nubs.
• Figure 5: Individual pixels rendered in Mistuba as independent BRDFs in Mistuba3. Input photograph to model with purple pixel indicating rendered BRDF (Left). Neural BRDF (Middle). Synthetic ground truth BRDF (Right).