Material Transforms from Disentangled NeRF Representations

TL;DR

A novel method for transferring material transformations across different scenes based on disentangled Neural Radiance Field representations is proposed, which learns to map Bidirectional Reflectance Distribution Functions (BRDF) from pairs of scenes observed in varying conditions.

Abstract

In this paper, we first propose a novel method for transferring material transformations across different scenes. Building on disentangled Neural Radiance Field (NeRF) representations, our approach learns to map Bidirectional Reflectance Distribution Functions (BRDF) from pairs of scenes observed in varying conditions, such as dry and wet. The learned transformations can then be applied to unseen scenes with similar materials, therefore effectively rendering the transformation learned with an arbitrary level of intensity. Extensive experiments on synthetic scenes and real-world objects validate the effectiveness of our approach, showing that it can learn various transformations such as wetness, painting, coating, etc. Our results highlight not only the versatility of our method but also its potential for practical applications in computer graphics. We publish our method implementation, along with our synthetic/real datasets on https://github.com/astra-vision/BRDFTransform

Figures (14)

• Figure 1: Overview of our proposed method. Our method takes observations of the same scene with two different materials $(\beta_0, \beta_1)$ for $s_0$ and $s_1$, respectively. We assume $\beta_1$ to be a function of $\beta_0$. Our method learns a joint representation and a transform function $\mathcal{F}_{}$ which maps the material of the first to the second (left block). Given a new scene $s$, we learn its geometry and material and apply our learned transform function (right block) to produce the same effects observed in the source scenes $(s_0, s_1)$.
• Figure 2: TensoIR on glossy surfaces. We observe that TensoIR overestimates roughness and smoothes the estimated illumination.
• Figure 3: Light estimation. We adopt a neural light representation liu2023nero which models direct and indirect light sources separately. On the indirect component, the two types of light sources are blended using an occlusion mask obtained via secondary ray casting along reflected light direction $t$jin2023tensoir. To avoid disrupting the optimization of the geometry, we reduce the gradient intensity along the directional inputs (on $n$ and $t$). We note $\bar{v}_t = 1 - v_t$, IDE is an Integrated Directional Encoding verbin2022refnerf while PE is a Positional Encoding mildenhall2020nerf.
• Figure 4: Synthetic dataset. Each column shows a difference scene $s^k, k \in \{\text{i}, \ldots, \text{viii}\}$. The first row shows the original scene, each subsequent row shows the scene after each synthetic transformation $T_j, j \in \{1, \ldots, 4\}$.
• Figure 5: Real-world dataset. Different bust figurines were first photographed with and without various colored coats (Beethoven, David, Schubert, Chopin, Wagner, Bach) or glossy varnishes (Mozart, Muse).