RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction

TL;DR

We address the challenge of reconstructing accurate 3D geometry from multi-view photos under non-Lambertian and complex illumination by unifying reflectance and normal maps within a neural volumetric rendering framework. The core idea is a pixel-wise re-parameterization that converts per-pixel reflectance and normals into a radiance vector under varying illumination, enabling a single objective optimization with a NeuS-like renderer. The method demonstrates state-of-the-art or competitive performance on the DiLiGenT-MV MVPS benchmark, with especially strong recovery of fine details in high-curvature and low-visibility regions, while remaining flexible to use with any PS technique. Although computationally intensive, the approach offers a principled, single-objective fusion of PS priors with neural rendering, and can benefit from faster NeuS2 implementations to enable practical use cases.

Abstract

This paper introduces a versatile paradigm for integrating multi-view reflectance (optional) and normal maps acquired through photometric stereo. Our approach employs a pixel-wise joint re-parameterization of reflectance and normal, considering them as a vector of radiances rendered under simulated, varying illumination. This re-parameterization enables the seamless integration of reflectance and normal maps as input data in neural volume rendering-based 3D reconstruction while preserving a single optimization objective. In contrast, recent multi-view photometric stereo (MVPS) methods depend on multiple, potentially conflicting objectives. Despite its apparent simplicity, our proposed approach outperforms state-of-the-art approaches in MVPS benchmarks across F-score, Chamfer distance, and mean angular error metrics. Notably, it significantly improves the detailed 3D reconstruction of areas with high curvature or low visibility.

Figures (13)

• Figure 1: One image from DiLiGenT-MV's Buddha dataset LiZWSDT20, and 3D reconstruction results from several recent MVPS methods: YangCCCW22KayaKOFG23mvpsnet and ours. The latter provides the fine details closest to the ground truth (GT), while being remarkably simpler.
• Figure 2: Overview of the proposed MVPS pipeline. The reflectance and normal maps provided for each view by PS are fused, by combining volume rendering with a pixel-wise re-parameterization of the inputs using physically-based rendering.
• Figure 3: High curvature (left) and low visibility (right) areas, on the Buddha and Reading datasets.
• Figure 4: Reconstructed 3D mesh and corresponding angular error of four objects from the DiLiGenT-MV benchmark.
• Figure 5: F-score (higher is better) as a function of the distance error threshold, in comparison with other state-of-the-art methods (a), and disabling individual components of our method (b).