Multi-view Surface Reconstruction Using Normal and Reflectance Cues

TL;DR

This work tackles high-fidelity 3D surface reconstruction under complex reflectance with sparse views by unifying normals and reflectance into radiance cues. It introduces a radiance-based re-parametrisation of per-pixel normals and reflectance, enabling seamless integration with both traditional MVS and neural volume rendering pipelines. Key contributions include an optimal illumination triplet, a reflectance-embedding technique to mitigate dark-region bias, an exact surface-sweeping variant for MVS, and RNb-NeuS2 with substantial speed-ups and robustness improvements evaluated on MVPS benchmarks (DiLiGenT-MV, LUCES-MV, Skoltech3D). The approach demonstrates state-of-the-art performance, especially in preserving fine geometry under challenging visibility and material conditions, and broadens MVPS applicability to broader MVS contexts such as DTU. Overall, the framework offers a practical, fast, and extensible MVPS pathway that leverages recent PS/NVR advancements to deliver detailed, relightable 3D reconstructions in semi-controlled environments.

Abstract

Achieving high-fidelity 3D surface reconstruction while preserving fine details remains challenging, especially in the presence of materials with complex reflectance properties and without a dense-view setup. In this paper, we introduce a versatile framework that incorporates multi-view normal and optionally reflectance maps into radiance-based surface reconstruction. Our approach employs a pixel-wise joint re-parametrization of reflectance and surface normals, representing them as a vector of radiances under simulated, varying illumination. This formulation enables seamless incorporation into standard surface reconstruction pipelines, such as traditional multi-view stereo (MVS) frameworks or modern neural volume rendering (NVR) ones. Combined with the latter, our approach achieves state-of-the-art performance on multi-view photometric stereo (MVPS) benchmark datasets, including DiLiGenT-MV, LUCES-MV and Skoltech3D. In particular, our method excels in reconstructing fine-grained details and handling challenging visibility conditions. The present paper is an extended version of the earlier conference paper by Brument et al. (in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024), featuring an accelerated and more robust algorithm as well as a broader empirical evaluation. The code and data relative to this article is available at https://github.com/RobinBruneau/RNb-NeuS2.

Figures (16)

• Figure 1: Overview of the surface reconstruction pipeline "RNb-NeuS" proposed in brument24rnbneus. From multi-view multi-light data, photometric stereo (noted PS) first estimates per-view normal maps $\{\mathbf{n}_{k}\}$ and optionally reflectance maps $\{\mathbf{r}_{k}\}$. These estimates are re-parametrised into radiance vectors $\mathbf{v}_{k} = \mathsf{F}(\mathbf{n}_{k}, \mathbf{r}_{k}, \mathsf{L}_{k})$ via a physically-based rendering function $\mathsf{F}$ under simulated lightings $\mathsf{L}_{k}$. An implicit neural representation, consisting of a signed distance function $f$ and a reflectance $\boldsymbol{\rho}$, is then learned by minimising the discrepancy between $\mathbf{v}_{k}$ and its volume-rendered counterpart $\tilde{\mathbf{v}}_k(f,\boldsymbol{\rho})$. The final surface mesh is extracted as the zero-level set of $f$ after optimisation.
• Figure 2: The three types of patches. From left to right: fronto-parallel patches, slanted patches and (proposed) surface patches.
• Figure 3: Synthetic normals (top) and reflectance (bottom) used in our experiments, for the reference view (left) and one control view (right).
• Figure 4: Mean depth estimation error as a function of Gaussian noise added to input normal maps. The multi-objective approach is highly sensitive to the tuning of the hyperparameter $\mu$, while ours maintains stable results without such tuning.
• Figure 5: Mean depth estimation error as a function of Gaussian noise added to input normals, for patch-based MVS method employing fronto-parallel plane sweeping (dark blue), slanted plane sweeping (purple), and normal-aware surface sweeping (pink), as well as the volumetric rendering method RNb-NeuS brument24rnbneus (orange). Surface sweeping achieves exact reconstruction in the noiseless case, yet volumetric rendering is much more robust to high noise levels.