NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering

TL;DR

This paper addresses the challenge of non-distant lighting in physics-based inverse rendering by introducing a NeRF-based non-distant emitter that captures spatially varying illumination beyond traditional environment maps. The authors develop a hybrid rendering framework, integrate a NeRF emitter through differentiable rendering, and propose emitter importance sampling to reduce variance. A multi-stage optimization initializes NeRF-based lighting, followed by joint refinement of shape, material, and lighting, tested on real and synthetic datasets with near-field illumination. The results show more accurate lighting representation and improved reconstructions, establishing NeRF emitters as a universal component for non-distant lighting in inverse rendering.

Abstract

Physics-based inverse rendering enables joint optimization of shape, material, and lighting based on captured 2D images. To ensure accurate reconstruction, using a light model that closely resembles the captured environment is essential. Although the widely adopted distant environmental lighting model is adequate in many cases, we demonstrate that its inability to capture spatially varying illumination can lead to inaccurate reconstructions in many real-world inverse rendering scenarios. To address this limitation, we incorporate NeRF as a non-distant environment emitter into the inverse rendering pipeline. Additionally, we introduce an emitter importance sampling technique for NeRF to reduce the rendering variance. Through comparisons on both real and synthetic datasets, our results demonstrate that our NeRF-based emitter offers a more precise representation of scene lighting, thereby improving the accuracy of inverse rendering.

Figures (5)

• Figure 1: The region within the bounding box is modeled by surfaces and material, while the region outside is handled by NeRF to account for environmental lighting. The NeRF-synthesized illumination viewed from two shading points (red (a) and blue (b) dots) is visualized on the right.
• Figure 2: We generate importance sampling distributions by (1) creating a point cloud of the bright parts of the NeRF, (2) clustering it into Gaussian mixtures, and (3) projecting these Gaussians to vMFs at shading points (red (a) and blue (b) dots).
• Figure 3: Comparing the gradient images rendered by our emitter importance sampling and pure BSDF sampling.
• Figure 4: Comparison with the environment map baseline on synthetic datasets.
• Figure 5: Comparison with the environment map baseline on real datasets.