MLI-NeRF: Multi-Light Intrinsic-Aware Neural Radiance Fields

TL;DR

This work proposes MLI-NeRF, which integrates Multiple Light information in Intrinsic-aware Neural Radiance Fields, and generates pseudo-label images for reflectance and shading to guide intrinsic image decomposition without the need for ground truth data.

Abstract

Current methods for extracting intrinsic image components, such as reflectance and shading, primarily rely on statistical priors. These methods focus mainly on simple synthetic scenes and isolated objects and struggle to perform well on challenging real-world data. To address this issue, we propose MLI-NeRF, which integrates \textbf{M}ultiple \textbf{L}ight information in \textbf{I}ntrinsic-aware \textbf{Ne}ural \textbf{R}adiance \textbf{F}ields. By leveraging scene information provided by different light source positions complementing the multi-view information, we generate pseudo-label images for reflectance and shading to guide intrinsic image decomposition without the need for ground truth data. Our method introduces straightforward supervision for intrinsic component separation and ensures robustness across diverse scene types. We validate our approach on both synthetic and real-world datasets, outperforming existing state-of-the-art methods. Additionally, we demonstrate its applicability to various image editing tasks. The code and data are publicly available.

Figures (18)

• Figure 1: Given real-world images from ReNe dataset Toschi_2023_CVPR (a), our method learns the neural radiance fields that enable novel view synthesis and relighting (b), and intrinsic decomposition (c) simultaneously. Image editing applications (d) can also be employed, such as reflectance editing, reflectance editing plus relighting or shading editing (simulating two lights).
• Figure 2: Illustration of the Framework. In Stage 1, we introduce light position as input to extend NeRF for multi-light implicit representation (top left). Following Stage 1, three post-processing steps are applied to generate pseudo labels for reflectance and shading using the proposed physics-based pipeline (right). In Stage 2, we train the intrinsic-aware NeRF based on the model from Stage 1 and the pseudo labels from post-processing (bottom left).
• Figure 3: Illustration of the pseudo reflectance generation process in the post-processing.
• Figure 4: Qualitative Results on the Synthetic Dataset with all settings. The same GT reflectance applies across all settings, but GT shading differs due to varying light positions. Here for brevity, only the shading under the Random Lights setting is shown. Compared to other methods, our approach predicts the best reflectance and effectively handles cast shadows.
• Figure 5: Qualitative Results on the Real Object Dataset. (a) Our method compared with NRHints for the rendered image, and PIE-Net and Ordinal for intrinsic decomposition. (b) Our reflectance estimation for two different scenes, with zoomed-in views on the object hole and cast shadow area.