NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

TL;DR

NeRO addresses the challenging problem of reconstructing both geometry and BRDF for highly reflective objects from multiview RGB images under unknown illumination. It introduces a two-stage framework: Stage I reconstructs geometry by approximating direct and indirect lighting with split-sum shading and a dual MLP light model, while Stage II refines BRDF via Monte Carlo shading on fixed geometry. A novel light representation with occlusion-aware direct/indirect components and integrated directional encoding enables accurate reconstruction without object masks. Extensive experiments on synthetic and real datasets demonstrate superior geometry accuracy and relighting quality compared with state-of-the-art methods, highlighting NeRO's practical potential for realistic rendering of glossy materials.

Abstract

We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment. Multiview reconstruction of reflective objects is extremely challenging because specular reflections are view-dependent and thus violate the multiview consistency, which is the cornerstone for most multiview reconstruction methods. Recent neural rendering techniques can model the interaction between environment lights and the object surfaces to fit the view-dependent reflections, thus making it possible to reconstruct reflective objects from multiview images. However, accurately modeling environment lights in the neural rendering is intractable, especially when the geometry is unknown. Most existing neural rendering methods, which can model environment lights, only consider direct lights and rely on object masks to reconstruct objects with weak specular reflections. Therefore, these methods fail to reconstruct reflective objects, especially when the object mask is not available and the object is illuminated by indirect lights. We propose a two-step approach to tackle this problem. First, by applying the split-sum approximation and the integrated directional encoding to approximate the shading effects of both direct and indirect lights, we are able to accurately reconstruct the geometry of reflective objects without any object masks. Then, with the object geometry fixed, we use more accurate sampling to recover the environment lights and the BRDF of the object. Extensive experiments demonstrate that our method is capable of accurately reconstructing the geometry and the BRDF of reflective objects from only posed RGB images without knowing the environment lights and the object masks. Codes and datasets are available at https://github.com/liuyuan-pal/NeRO.

Figures (32)

• Figure 1: We apply different multiview reconstruction methods to reconstructing reflective objects. (a) The input images. Reconstruction results of (b) COLMAP schoenberger2016mvs, (c) NeuS wang2021neus, (d) Ref-NeRF verbin2022ref, (e) NDRMC hasselgren2022shape and (f) our method. In images of column (a), we use red bounding boxes to indicate regions illuminated by indirect lights. $^*$NDRMC uses ground-truth object masks for training while all the other methods are trained without object masks.
• Figure 2: Specular lobe in Eq. \ref{['eq:app_spe']} is determined by the roughness $\rho$ and the reflective direction $\mathbf{t}$. (a) A smooth surface with small $\rho$ has a smaller specular lobe while (b) a rougher surface with large $\rho$ has a larger specular lobe.
• Figure 3: Direct and indirect lights for a point $\mathbf{p}$. The direct light in the direction $\omega_0$ is not occluded while the indirect light in the direction $\omega_1$ is occluded by surfaces inside the unit sphere. $s$ is the occlusion probability. $\mathbf{q}(\mathbf{p},\omega)$ is the intersection point on the bounding sphere of the ray emitting from $\mathbf{p}$ along the direction $\omega$.
• Figure 4: Effects of occlusion loss. (Top) Without $\ell_{\rm occ}$, the occlusion probability predicted by $g_{\rm occ}$ will be completely inconsistent with the reconstructed geometry and causes incorrect reconstruction. (Bottom) With $\ell_{\rm occ}$, the predicted occlusion probability is accurate and the reconstruction is correct.
• Figure 5: Architecture of networks in Stage I. "PE"is positional encoding mildenhall2020nerf while "IDE" and "DE" are integrated direction encoding verbin2022ref and vanilla directional encoding, respectively.