Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects

TL;DR

Factored-NeuS tackles the challenging problem of reconstructing geometry, materials, and illumination of glossy scenes from posed multi-view images without extra data. It introduces a three-stage progressive inverse rendering pipeline that combines volume and surface rendering, a continuous light-visibility model, and a learnable specular albedo within a BRDF framework to factorize surface, materials, and lighting. The approach achieves state-of-the-art surface geometry recovery on real datasets with complex highlights and demonstrates superior material and illumination decomposition against strong baselines, enabling reliable relighting and editing. These results advance implicit-surface inverse rendering by integrating glossy handling into a data-driven, stage-wise optimization that remains robust to real-world lighting and material variability.

Abstract

We develop a method that recovers the surface, materials, and illumination of a scene from its posed multi-view images. In contrast to prior work, it does not require any additional data and can handle glossy objects or bright lighting. It is a progressive inverse rendering approach, which consists of three stages. In the first stage, we reconstruct the scene radiance and signed distance function (SDF) with a novel regularization strategy for specular reflections. We propose to explain a pixel color using both surface and volume rendering jointly, which allows for handling complex view-dependent lighting effects for surface reconstruction. In the second stage, we distill light visibility and indirect illumination from the learned SDF and radiance field using learnable mapping functions. Finally, we design a method for estimating the ratio of incoming direct light reflected in a specular manner and use it to reconstruct the materials and direct illumination. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art in recovering surfaces, materials, and lighting without relying on any additional data.

Figures (26)

• Figure 1: Left: Geometry visualization for NeuS, Geo-NeuS and our method on the Pot scene from SK3D dataset. Existing surface reconstruction methods struggle to recover the correct geometry of glossy objects due to the complex view-dependent effects they induce. The weak color model of these methods compels to represent such effects through concave geometric deformations rather than proper view-dependent radiance, leading to shape artifacts. In contrast, our method can correctly reconstruct a highly reflective surface due to our joint appearance, diffuse, and specular color training strategy. Right: Visualization of the recovered diffuse color component on the Bunny scene from DTU for IndiSG Indirect and our method. Existing inverse rendering methods overestimate the diffuse material component in the presence of specular highlights. Our regularization strategy allows us to properly disentangle the color into diffuse and specular components.
• Figure 2: Overview for Stage 1 (left), Stage 2 (mid), and Stage 3 (right) training pipelines. The first stage (left) trains the SDF network $\mathsf{S}_\theta$ which outputs a feature vector $\bm{v}_f \in \mathbb{R}^{256}$, SDF value $s \in \mathbb{R}$, and normal $\bm{n} \in \mathbb{S}^2$ (as a normalized gradient of $s$; denoted via the dashed line); diffuse and specular surface color networks $\mathsf{M}_d$ and $\mathsf{M}_s$ produce their respective colors $\bm{c}_d, \bm{c}_s \in \mathbb{R}^3$ via surface rendering, which are then combined through tone mapping $\gamma(\cdot)$ to get the final surface color ${C}^\text{sur} \in \mathbb{R}^3$; volumetric color network $\mathsf{M}_c$ produces the volumetrically rendered color ${C}^\text{vol} \in \mathbb{R}^3$. The $\mathsf{ref}$ operation denotes computation of the reflection direction $\boldsymbol{\omega}_r \in \mathbb{S}^2$ from normal $\bm{n}$ and ray direction $\boldsymbol{\omega} \in \mathbb{S}^2$. In the second stage (mid), indirect light network $\mathsf{M}_\text{ind}$ and light visibility network $\mathsf{M}_\nu$ produce their respective indirect light SGs and light visibility $\nu$. In the third stage (right), we optimize the material BRDF auto-encoder with the sparsity constraint Indirect, our novel specular albedo network $\mathsf{M}_{sa}$, and the indirect illumination network $\mathsf{M}_\text{ind}$. See Sec \ref{['method']} for details.
• Figure 3: Qualitative results for DTU (left) and SK3D (right).
• Figure 4: Qualitative results for the Shiny dataset(toaster and coffee).
• Figure 5: Qualitative results for Glossy dataset.