Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields

TL;DR

This work proposes a novel 5D Neural Plenoptic Function (NeP) based on NeRFs and ray tracing, such that more accurate lighting-object interactions can be formulated via the ren-dering equation, and designs a material-aware cone sampling strategy to efficiently integrate lights inside the BRDF lobes with the help of pre-filtered radiance fields.

Abstract

Inverse rendering aims at recovering both geometry and materials of objects. It provides a more compatible reconstruction for conventional rendering engines, compared with the neural radiance fields (NeRFs). On the other hand, existing NeRF-based inverse rendering methods cannot handle glossy objects with local light interactions well, as they typically oversimplify the illumination as a 2D environmental map, which assumes infinite lights only. Observing the superiority of NeRFs in recovering radiance fields, we propose a novel 5D Neural Plenoptic Function (NeP) based on NeRFs and ray tracing, such that more accurate lighting-object interactions can be formulated via the rendering equation. We also design a material-aware cone sampling strategy to efficiently integrate lights inside the BRDF lobes with the help of pre-filtered radiance fields. Our method has two stages: the geometry of the target object and the pre-filtered environmental radiance fields are reconstructed in the first stage, and materials of the target object are estimated in the second stage with the proposed NeP and material-aware cone sampling strategy. Extensive experiments on the proposed real-world and synthetic datasets demonstrate that our method can reconstruct high-fidelity geometry/materials of challenging glossy objects with complex lighting interactions from nearby objects. Project webpage: https://whyy.site/paper/nep

Figures (7)

• Figure 1: Inverse rendering results of the cutting-edge method, NeRO nero, and ours from calibrated multi-view images of a glossy object. Geometries are shown as a rendered mesh in the second and third images, and materials (metalness & roughness) are shown as a color map in the fourth and fifth images. We can see that our results not only have smoother and more accurate geometry but also present a more reasonable material (since the material of this object should be uniform).
• Figure 2: The pipeline of the proposed method. Our method has two stages: the fields learning stage for object geometry reconstruction and the neural radiance fields optimization, and the material learning stage using ray tracing.
• Figure 3: The detailed structure of our proposed method. Fields learning stage consists of an SDF-based object field and a Mip-NeRF as the environmental field. Based on them, we construct our neural plenoptic function via ray tracing and material-aware cone sampling method to represent the global illumination.
• Figure 4: Comparison of the geometry reconstruction among cutting-edge methods and ours. For each method, we utilize marching cubes to extract the triangle meshes for comparison.
• Figure 5: Samples from the proposed dataset. The first two examples are synthetic and the last two samples are real-world captured.