Learning Relighting and Intrinsic Decomposition in Neural Radiance Fields

TL;DR

This paper tackles the problem of learning relighting and intrinsic decomposition in neural radiance fields for real scenes with backgrounds, without relying on ground-truth intrinsic data. It introduces a two-stage approach: Stage 1 trains a relightable NeRF-like scene representation to render novel views under varying illumination, and Stage 2 uses physics-based pseudo labels derived from the Stage 1 results to supervise intrinsic decomposition into reflectance and shading, with a residual term to capture non-Lambertian effects. Pseudo labels for shading and reflectance are generated via normals and light visibility (from sphere tracing), a gamma-corrected shading model, and multi-illumination fusion using K-means, all underpinned by a Lambertian-plus-residual formulation I(i,j) = R(i,j) ⊙ S(i,j) + Re(i,j). The method achieves convincing relighting and intrinsic decomposition on synthetic and real datasets, enabling editing tasks like reflectance and shading adjustments while reducing dependence on pre-trained priors or GT. This work contributes a practical, physics-grounded framework for 3D-consistent intrinsic editing in NeRFs with backgrounds, with potential extensions to more diverse scenes.

Abstract

The task of extracting intrinsic components, such as reflectance and shading, from neural radiance fields is of growing interest. However, current methods largely focus on synthetic scenes and isolated objects, overlooking the complexities of real scenes with backgrounds. To address this gap, our research introduces a method that combines relighting with intrinsic decomposition. By leveraging light variations in scenes to generate pseudo labels, our method provides guidance for intrinsic decomposition without requiring ground truth data. Our method, grounded in physical constraints, ensures robustness across diverse scene types and reduces the reliance on pre-trained models or hand-crafted priors. We validate our method on both synthetic and real-world datasets, achieving convincing results. Furthermore, the applicability of our method to image editing tasks demonstrates promising outcomes.

Figures (14)

• Figure 1: Given images under different camera pose and light positions (a), our method learns the neural radiance fields that enable novel view synthesis and relighting (b), and intrinsic decomposition (c) simultaneously. Other image editing applications (d), such as reflectance editing, reflectance editing + relighting, and shading editing (simulating two lights), can also be employed.
• Figure 2: Method Framework: Stage 1 involves learning the neural field with relighting (top left). Post-processing and generating pseudo labels (right). In Stage 2, the learning process continues to learn intrinsic decomposition based on the model trained in Stage 1 and the pseudo labels (bottom left).
• Figure 3: Qualitative comparisons on the NeRF mildenhall2021nerf (the first two rows) and ReNe Toschi_2023_CVPR datasets (the last two rows).
• Figure 4: Post-processing and generating pseudo labels.
• Figure 5: Compare with other methods on the ReNe dataset.