PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields

TL;DR

This work addresses a key limitation in most NeRF and 3D Gaussian Splatting approaches: they estimate view-dependent appearance without modeling scene materials and illumination, and presents an inverse rendering (IR) model capable of jointly estimating scene geometry, materials, and illumination.

Abstract

We tackle the ill-posed inverse rendering problem in 3D reconstruction with a Neural Radiance Field (NeRF) approach informed by Physics-Based Rendering (PBR) theory, named PBR-NeRF. Our method addresses a key limitation in most NeRF and 3D Gaussian Splatting approaches: they estimate view-dependent appearance without modeling scene materials and illumination. To address this limitation, we present an inverse rendering (IR) model capable of jointly estimating scene geometry, materials, and illumination. Our model builds upon recent NeRF-based IR approaches, but crucially introduces two novel physics-based priors that better constrain the IR estimation. Our priors are rigorously formulated as intuitive loss terms and achieve state-of-the-art material estimation without compromising novel view synthesis quality. Our method is easily adaptable to other inverse rendering and 3D reconstruction frameworks that require material estimation. We demonstrate the importance of extending current neural rendering approaches to fully model scene properties beyond geometry and view-dependent appearance. Code is publicly available at https://github.com/s3anwu/pbrnerf

Figures (7)

• Figure 1: Improving inverse rendering with physics-based priors. The proposed PBR-NeRF significantly outperforms the NeILF++ zhang2023neilf++ baseline simply by using our novel Conservation of Energy and NDF-weighted Specular Losses. Our physics-based losses correct "baked-in" specular highlights misrepresented in the diffuse lobe (highlighted areas) by (1) enforcing energy conservation and (2) accurately separating specular and diffuse reflections. The result is a more realistic, physically consistent material and lighting estimation.
• Figure 2: Overview of our PBR-NeRF architecture for neural inverse rendering. Our two novel physics-based material losses, i.e. $\mathcal{L}_{\text{cons}}$ for conservation of energy and $\mathcal{L}_{\text{spec}}$ for disentangling the diffuse and specular BRDF components, are derived in Sec. \ref{['sec:pbr_losses']}. The complete PBR-NeRF model comprises multiple neural fields, which are optimized in a stage-wise fashion: a standard NeRF+SDF modeling radiance and geometry, a neural incident light field (NeILF) modeling spatially-varying illumination, and a BRDF field modeling materials via the Disney BRDF burley2012physically. Our physics-based losses provide the BRDF field with valuable inductive biases for improved material estimation, which help resolve to a large extent the inherent material-lighting ambiguity in inverse rendering and thus benefit the incident light field as well. Consequently, novel scene views synthesized with our model enjoy state-of-the-art quality.
• Figure 3: Illustration of our physics-based losses. We constrain our Disney BRDF burley2012physically material and NeILF yao2022neilfzhang2023neilf++ incident light estimation with two novel physics-based losses: (1) the Conservation of Energy Loss $\mathcal{L}_\text{cons}$ to supervise the complete BRDF $f_r = f_s + f_d$ denoted by the dotted envelope, and (2) the NDF-weighted Specular Loss $\mathcal{L}_\text{spec}$ to adjust the relative magnitudes of the specular $f_s$ (red) and diffuse $f_d$ (blue) BRDF lobes.
• Figure 4: Qualitative comparisons on the NeILF++ dataset zhang2023neilf++. †: no ground-truth environment maps are provided with the dataset.
• Figure 5: Qualitative comparisons on DTU jensen2014dtu. Best viewed on a screen at full zoom.