Relighting Scenes with Object Insertions in Neural Radiance Fields

TL;DR

This work introduces a novel pipeline based on Neural Radiance Fields (NeRFs) for seamlessly integrating objects into scenes, from two sets of images depicting the object and scene, and introduces an innovative approach for efficient shadow rendering by comparing the depth maps between the camera view and the light source view and generating vivid soft shadows.

Abstract

The insertion of objects into a scene and relighting are commonly utilized applications in augmented reality (AR). Previous methods focused on inserting virtual objects using CAD models or real objects from single-view images, resulting in highly limited AR application scenarios. We propose a novel NeRF-based pipeline for inserting object NeRFs into scene NeRFs, enabling novel view synthesis and realistic relighting, supporting physical interactions like casting shadows onto each other, from two sets of images depicting the object and scene. The lighting environment is in a hybrid representation of Spherical Harmonics and Spherical Gaussians, representing both high- and low-frequency lighting components very well, and supporting non-Lambertian surfaces. Specifically, we leverage the benefits of volume rendering and introduce an innovative approach for efficient shadow rendering by comparing the depth maps between the camera view and the light source view and generating vivid soft shadows. The proposed method achieves realistic relighting effects in extensive experimental evaluations.

Figures (17)

• Figure 1: We propose a method for inserting object NeRFs into scene NeRFs. From left to right, the figure depicts input images, object insertion without relighting, and relightings from novel views. The method also supports non-Lambertian relighting and material rendering, as the pillow on the chair.
• Figure 2: Overview of the method. Given two sets of multi-view images of a scene and an object from unknown lighting environments, our method reconstructs and predicts the intrinsic decomposition of the scene and object. Subsequently, it composites their geometry through segmented sampling, updating the object's shading to match the scene illumination. At last, non-Lambertian relighting of the composited scene is proposed with the hybrid lighting representation and efficient shadow mapping.
• Figure 3: Visualizations comparing cases with (on the right) and without (on the left) the term $\mathcal{L}_d$. This row displays illustrative examples from a scene.
• Figure 4: Visualization of the rendered images depicting the segmented sampling strategy. Part of the Lego is missing on the left image.
• Figure 5: Variance shadow mapping. By comparing depth inconsistencies from the light view and the camera view, we have efficient shadow mapping for rendering cast shadows. For example, depths of P1 and P4 are consistent under two views, then they are visible. Depths of P2 are inconsistent in two views, so it is occluded, i.e. in shadows. In the right, blue areas are hard shadows by naive shadow mapping, where P4 is not in shadows. By variance shadow mapping, P4 is in soft shadows with visibility in $[0,1]$.