NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination

TL;DR

NeRFactor addresses the challenge of recovering shape and spatially varying reflectance from multi-view images captured under one unknown illumination by combining NeRF-based geometry initialization with a joint optimization over surface normals, light visibility, albedo, BRDF latent codes, and environment lighting. It advances inverse rendering by explicitly modeling visibility and shadows and by introducing a data-driven BRDF prior learned from real measurements, enabling realistic relighting with arbitrary lighting and material editing. The approach yields high-quality geometry suitable for free-viewpoint relighting, supports both synthetic and real scenes, and demonstrates robust performance even when starting from MVS geometry. Together, these innovations move toward robust 3D asset recovery from casual captures, with practical impact on relighting, material editing, and view synthesis in uncontrolled settings.

Abstract

We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.

Figures (12)

• Figure 1: NeRFactor is a coordinate-based model that factorizes, in an unsupervised manner, the appearance of a scene observed under one unknown lighting condition. It tackles this severely ill-posed problem by using a reconstruction loss, simple smoothness regularization, and data-driven BRDF priors. Modeling visibility explicitly, NeRFactor is a physically-based model that supports shadows under arbitrary lighting.
• Figure 2: High-quality geometry recovered by NeRFactor. (A) We can directly derive the surface normals and light visibility from a trained NeRF. However, geometry derived in this way is too noisy to be used for relighting (see https://www.youtube.com/watch?v=UUVSPJlwhPg). (B) Jointly optimizing shape and reflectance improves the NeRF geometry, but there is still significant noise (e.g., the stripe artifacts in II). (C) Joint optimization with smoothness constraints leads to smooth surface normals and light visibility that resemble ground truth. Visibility averaged over all incoming light directions is ambient occlusion.
• Figure 3: Joint optimization of shape, reflectance, and lighting. Although our recovered surface normals, visibility, and albedo sometimes omit some fine-grained detail, they still closely resemble the ground truth. Despite that lighting recovered by NeRFactor is heavily oversmoothed (because our objects are not shiny) and incorrect on the bottom half of the hemisphere (since objects are only ever observed from the top hemisphere), the dominant light sources and occluders are localized nearby their ground-truth locations in the light probes. Note that we are unable to compare against ground-truth BRDFs, as they are defined using Blender's shader node trees, while our recovered BRDFs are parameterized by our learned model.
• Figure 4: Free-viewpoint relighting. The factorization that NeRFactor produces can be used to perform "free-viewpoint relighting": rendering a novel view of the object under arbitrary lighting conditions including the challenging OLAT conditions. The renderings produced by NeRFactor qualitatively resemble the ground truth and accurately exhibit challenging effects such as specularities and cast shadows (both hard and soft).
• Figure 5: Results of real-world captures. (I) Given images of a real-world object lit by unknown lighting (A), NeRFactor factorizes its appearance into albedo (C), spatially-varying BRDF latent codes (D), surface normals (E), and light visibility for all incoming light directions (visualized here as ambient occlusion; F). Note how the estimated flower albedo is shading-free. (II) With this factorization, one can synthesize novel views of the scene relit by any arbitrary lighting. Even on these challenging real-world scenes, NeRFactor is able to synthesize realistic specularities and shadows across various lighting conditions.