REF$^2$-NeRF: Reflection and Refraction aware Neural Radiance Field

TL;DR

This work tackles the challenge of reconstructing scenes containing glass by explicitly modeling refraction and reflection. It introduces a glass network to predict refraction points and offsets, coupled with a Decomposition NeRF that separately handles view-independent and view-dependent components, enabling explicit separation of direct and reflected light. The approach blends these components to render novel views and is trained with known camera poses obtained via a robotic arm, showing improved modeling of refraction surfaces and reflections on both simulated and real datasets. Results indicate robust performance in separating viewpoint-dependent reflections from static content and accurate estimation of glass surfaces, offering practical improvements for scenes with transparent enclosures.

Abstract

Recently, significant progress has been made in the study of methods for 3D reconstruction from multiple images using implicit neural representations, exemplified by the neural radiance field (NeRF) method. Such methods, which are based on volume rendering, can model various light phenomena, and various extended methods have been proposed to accommodate different scenes and situations. However, when handling scenes with multiple glass objects, e.g., objects in a glass showcase, modeling the target scene accurately has been challenging due to the presence of multiple reflection and refraction effects. Thus, this paper proposes a NeRF-based modeling method for scenes containing a glass case. In the proposed method, refraction and reflection are modeled using elements that are dependent and independent of the viewer's perspective. This approach allows us to estimate the surfaces where refraction occurs, i.e., glass surfaces, and enables the separation and modeling of both direct and reflected light components. The proposed method requires predetermined camera poses, but accurately estimating these poses in scenes with glass objects is difficult. Therefore, we used a robotic arm with an attached camera to acquire images with known poses. Compared to existing methods, the proposed method enables more accurate modeling of both glass refraction and the overall scene.

Figures (8)

• Figure 1: Image acquisition system and example images of a scene including a glass case and objects. Images of those scenes contain effects of light ray reflection and refraction, which vary depending on viewpoint.
• Figure 2: Overview of the proposed framework. Glass network MLP models refraction occurred by transparent object and adjusted each sampled position. Then, we decompose the scene into view-dependent and view-independent components to separate reflection from input images and model both.
• Figure 3: Network architecture of the proposed method. The glass network outputs the glass density and offset, which modify the ray by the refraction effect through glass walls. Here, the glass density is view-independent, and the offset is view-dependent. The NeRF network takes the adjusted position as input and outputs the view-independent and view-dependent densities and color or feature. The feature renderer provides the corresponding feature map, and the decoder and gate MLPs convert the rendered feature map to a view-dependent image with a blending weight. Finally, the image blending module generates the image by composing the rendered view-independent and view-dependent images. The training process minimizes the loss calculated from the composed image and the input image while optimizing the MLPs.
• Figure 4: Structure of the proposed method to express light refraction as volume rendering using glass density to estimate the offset. Here, refraction is simplified as a parallel translation in 3D space occurring on the glass surface. We estimate the path of the light considering refraction by accumulating the vectors of this translation.
• Figure 5: A proposal method structure that divides the scene into two fields. Elements that do not change depending on the viewpoint, e.g., objects and backgrounds in the scene, are represented in the view-independent field. Elements that do change depending on the viewpoint, e.g., reflections caused by glass and reflections from light sources, are represented in the view-dependent field, where the density changes based on the viewpoint.