NeuS-PIR: Learning Relightable Neural Surface using Pre-Integrated Rendering

TL;DR

This work introduces NeuS-PIR, a method for recovering relightable neural surfaces from multi-view imagery by jointly learning geometry, material properties, and illumination through differentiable pre-integrated rendering. It builds on NeuS for explicit-like geometry and adds a dual-branch radiance model that factorizes the radiance field into a spatially varying material field and an all-frequency lighting representation stored as a high-frequency environment cubemap, with regularization to stabilize training. An indirect illumination distillation pipeline further captures inter-reflection and other complex lighting effects, enabling realistic relighting and integration with modern graphics engines. Extensive experiments on synthetic and real datasets show NeuS-PIR outperforms relevant baselines in relighting quality, albedo accuracy, and geometric fidelity, while ablations confirm the importance of each component (geometry, material, illumination, and indirect lighting) in achieving high-quality inverse rendering.

Abstract

This paper presents a method, namely NeuS-PIR, for recovering relightable neural surfaces using pre-integrated rendering from multi-view images or video. Unlike methods based on NeRF and discrete meshes, our method utilizes implicit neural surface representation to reconstruct high-quality geometry, which facilitates the factorization of the radiance field into two components: a spatially varying material field and an all-frequency lighting representation. This factorization, jointly optimized using an adapted differentiable pre-integrated rendering framework with material encoding regularization, in turn addresses the ambiguity of geometry reconstruction and leads to better disentanglement and refinement of each scene property. Additionally, we introduced a method to distil indirect illumination fields from the learned representations, further recovering the complex illumination effect like inter-reflection. Consequently, our method enables advanced applications such as relighting, which can be seamlessly integrated with modern graphics engines. Qualitative and quantitative experiments have shown that NeuS-PIR outperforms existing methods across various tasks on both synthetic and real datasets. Source code is available at https://github.com/Sheldonmao/NeuSPIR

Figures (12)

• Figure 1: In our proposed method, we achieve simultaneous learning of geometry, material, and illumination within the neural implicit field. The results, displayed in the bottom row, demonstrate significant improvements compared to NVDiffrec munkberg2022extracting showcased in the top row. Our method excels in relighting the image and reconstructing geometry. By building upon NeuS wang2021neus, a popular approach for geometry reconstruction without factorization, our mesh incorporates the advantages of material and illumination learning, resulting in enhanced geometry preservation, particularly in highly reflective areas.
• Figure 2: In our approach, we decompose a scene into three components: geometry, material, and illumination. To visualize the results, we present the reference and predicted environment illuminations as latitude-longitude converted environment cubemaps. The roughness and metallic properties are visualized using a jet color map, which represents values ranging from 0 to 1.
• Figure 3: The network architecture of our proposed method consists of multiple components. The SDF MLP is responsible for learning the geometry, while the radiance MLP focuses on capturing the radiance field at a coarse level. Through additional decompositions, we employ the Material MLP to factorize the material properties, and the Pre-Integrated Light module to handle illumination aspects. Together, these components contribute to the overall functionality and capabilities of our method.
• Figure 4: Novel view synthesis and relighting results produced by our proposed method.
• Figure 5: An example of material factorization by our proposed method.