RRM: Relightable assets using Radiance guided Material extraction

TL;DR

RRM tackles relighting of NeRF-like scenes under arbitrary lighting by learning four interdependent components: a density field for geometry, a physically-inspired radiance field, a material field, and a Laplacian-Pyramid–based environment lighting. A radiance module splits radiance into view-dependent $c_d$ and view-independent $c_i$ terms and uses directional encoding with $\omega_r$ and IDE to improve normals for glossy materials. A MIS-based PBR pathway renders novel views by integrating diffuse and specular terms from learned $\gamma_x$, $F_{0,x}$, $\rho_x$, and $\kappa$, with environment lighting captured by the Laplacian Pyramid. Experiments demonstrate improved normal reconstruction and competitive relighting against NMF and TensoIR on synthetic datasets, with ablations confirming the impact of radiance decomposition, $\kappa \rightarrow \rho$ mapping, and PoL lighting, underscoring the practical potential of end-to-end radiance-guided inverse rendering for glossies and complex geometries.

Abstract

Synthesizing NeRFs under arbitrary lighting has become a seminal problem in the last few years. Recent efforts tackle the problem via the extraction of physically-based parameters that can then be rendered under arbitrary lighting, but they are limited in the range of scenes they can handle, usually mishandling glossy scenes. We propose RRM, a method that can extract the materials, geometry, and environment lighting of a scene even in the presence of highly reflective objects. Our method consists of a physically-aware radiance field representation that informs physically-based parameters, and an expressive environment light structure based on a Laplacian Pyramid. We demonstrate that our contributions outperform the state-of-the-art on parameter retrieval tasks, leading to high-fidelity relighting and novel view synthesis on surfacic scenes.

Figures (14)

• Figure 1: We take as input a collection of photographs from a scene, and extract a model with physically-based parameters from which we can set a new lighting condition. In comparison to NMF mai2023neural and TensoIR jin2023tensoir, our method is more robust to glossy materials and better handles self-reflection, as it is able to reconstruct more accurate surface normals.
• Figure 2: Overview of our model. At each ray-marching step, we evaluate a density $\sigma$, which weights physically-based material properties and radiance information as we integrate these quantities along the marched ray. Physically-based properties are processed by our PBR fixed module to compute the final radiance. Grey boxes are fixed functions, while colored boxes are the learnable scene representation. Orange boxes and arrows are used for supervision only and dropped when evaluating with a new environment lighting. See other figures for zooms of each component.
• Figure 3: Our radiance component decodes the latent appearance vector coming from the material component into view-dependent and view-independent (diffuse) terms. This is done using two isolated neural networks, only one of which receives the view direction as input. The view-dependent network is made more robust to reflections by using a directional encoding based on the prediction of the material component. A drop-out on the view-dependent term ensures that the diffuse term gets as much magnitude as possible.
• Figure 4: Visualization of our radiance decomposition as described in Fig. \ref{['fig:radiance-component']} after overfitting on the helmet scene. This qualitatively corresponds to the diffuse and specular terms of a PBR BSDF model.
• Figure 5: The material component of our scene representation consists of a look-up in a TensoRF $\mathcal{G}_a$ followed by a simple neural network to decode the sampled latent appearance vector into physically-based material properties. This two-stage approach enables the radiance-based component to guide the definition of a latent appearance without learning a full mapping from physically-based parameters.