Planar Reflection-Aware Neural Radiance Fields

TL;DR

A planar reflection-aware NeRF is introduced that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections, and obtains accurate scene geometry.

Abstract

Neural Radiance Fields (NeRF) have demonstrated exceptional capabilities in reconstructing complex scenes with high fidelity. However, NeRF's view dependency can only handle low-frequency reflections. It falls short when handling complex planar reflections, often interpreting them as erroneous scene geometries and leading to duplicated and inaccurate scene representations. To address this challenge, we introduce a reflection-aware NeRF that jointly models planar reflectors, such as windows, and explicitly casts reflected rays to capture the source of the high-frequency reflections. We query a single radiance field to render the primary color and the source of the reflection. We propose a sparse edge regularization to help utilize the true sources of reflections for rendering planar reflections rather than creating a duplicate along the primary ray at the same depth. As a result, we obtain accurate scene geometry. Rendering along the primary ray results in a clean, reflection-free view, while explicitly rendering along the reflected ray allows us to reconstruct highly detailed reflections. Our extensive quantitative and qualitative evaluations of real-world datasets demonstrate our method's enhanced performance in accurately handling reflections.

Figures (8)

• Figure 5: Sparse edge regularization. By applying the sparse edge regularization, the primary view is clean and free of false geometries.
• Figure 6: Individual component. Rendering along the primary ray yields a reflection-free image. Rendering along the reflected ray captures the source of the reflection. The final image is composed by combining the reflection-free image with the attenuated source of the reflection. Our method ensures that reflections are effectively isolated and can be manipulated independently from the reflection-free scene reconstruction.
• Figure 7: Accurate geometry. By explicitly casting reflected rays, our method ensures accurate geometry representation, which is not degraded by the false geometries introduced by reflections.
• Figure 8: Reflection-removal comparison on the outside split. We evaluate the reflection-removal performance of each method using held-out views captured outside the rooms. As the query viewpoints outside the room are far from the training views (inside the room), artifacts are thus visible.
• Figure 9: Reflection-removal comparison on the inside-val split. We qualitatively evaluate the reflection-removal performance of each method using held-out views captured inside the rooms.