Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields

TL;DR

This paper tackles the difficulty of reproducing glossy reflections in Neural Radiance Fields by reparameterizing outgoing radiance as a function of the reflection direction about local normals. It introduces Ref-NeRF, which incorporates Integrated Directional Encoding and a diffuse/specular decomposition to enable smoother, more accurate interpolation of view-dependent appearance. A normal-regularizer and predicted normals improve reflection directions, yielding sharper specular highlights and more faithful geometry, while the structured radiance representation supports interpretable scene editing. Empirically, Ref-NeRF achieves state-of-the-art renderings on challenging glossy scenes and real-world captures, with notable improvements in both rendering quality and normal accuracy over mip-NeRF and related baselines, at the cost of modest additional computation.

Abstract

Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multilayer perceptrons that provide the volume density and view-dependent emitted radiance at each location. While NeRF-based techniques excel at representing fine geometric structures with smoothly varying view-dependent appearance, they often fail to accurately capture and reproduce the appearance of glossy surfaces. We address this limitation by introducing Ref-NeRF, which replaces NeRF's parameterization of view-dependent outgoing radiance with a representation of reflected radiance and structures this function using a collection of spatially-varying scene properties. We show that together with a regularizer on normal vectors, our model significantly improves the realism and accuracy of specular reflections. Furthermore, we show that our model's internal representation of outgoing radiance is interpretable and useful for scene editing.

Figures (10)

• Figure 1: Ref-NeRF significantly improves normal vectors (top row) and visual realism (remaining rows) compared to mip-NeRF, the previous top-performing neural view synthesis model. Ref-NeRF's improvements are apparent in rendered frames (Rows 2 & 3), and even more in rendered videos (bottom row epipolar plane images and supplementary video), where its glossy highlights shift realistically across views instead of blurring and fading like mip-NeRF's. Image PSNR (higher is better) and surface normal mean angular error (lower is better) shown as insets.
• Figure 2: Visualizations of outgoing radiance in NeRF and Ref-NeRF, using 2D position-angle slices of radiance along an $x$-parameterized surface curve on a glossy object under colored lights. Because NeRF (middle row) uses view angle $\phi_o$ as input, when presented with glossy reflectances (left) or spatially-varying materials (right) it must interpolate between highly complicated functions like the irregularly curved colored lines shown here. In contrast, Ref-NeRF (bottom row) parameterizes radiance using a normal vector $\hat{\mathbf{n}}$ and a reflection angle $\phi_r$, and adds diffuse color $\mathbf{c}_{d}$ and roughness $\rho$ to its spatial MLP, which collectively makes radiance functions simple to model even for shiny or spatially-varying materials. The gray checkerboard indicates directions below the surface at position $x$.
• Figure 3: We enable the directional MLP to represent reflected radiance functions for any continuously-valued roughness using our integrated directional encoding. Each component of the encoding is a spherical harmonic function convolved with a vMF distribution with concentration parameter $\kappa$, output by our spatial MLP (equivalent to the expectation of the spherical harmonic under the vMF). Less rough locations receive higher-frequency encodings (top), while more rough regions receive encodings with attenuated high frequencies. Our IDE allows lighting information to be shared between locations with different roughnesses, and lets reflectance be edited.
• Figure 4: A visualization of mip-NeRF's and our architectures.
• Figure 5: Prior top-performing NeRF-based approaches can fail catastrophically in highly-reflective scenes. Mip-NeRF (right column) produces blurry renderings of reflections that are inconsistent over different views (see EPI), and does not correctly simulate the appearance of the two different surface roughnesses. Our model (middle column) reconstructs the object almost perfectly. The accumulated normals and rendering weights $w_i$ along the central scanline of the image (bottom row) show that mip-NeRF mimics specularities using emitters inside the object, while Ref-NeRF correctly recovers a concentrated surface.

Theorems & Definitions (6)

• Claim 1
• proof
• Claim 2
• proof
• Claim 3
• proof