NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images

TL;DR

This work tackles the challenging problem of reconstructing 3D geometry for reflective objects from sparse polarized views. It introduces NeRSP, a neural implicit (SDF-based) reconstruction framework that jointly exploits geometric cues from AoP-derived azimuth maps and photometric cues from a polarimetric image formation model to constrain shape under sparse inputs. The method predicts diffuse radiance and surface roughness to render Stokes vectors along rays, enabling end-to-end optimization that yields accurate shapes with as few as six views, outperforming RGB-based and polarization-only baselines. To support rigorous evaluation, the authors present RMVP3D, a real-world multiview polarized dataset with aligned ground-truth meshes, and demonstrate state-of-the-art results on both synthetic and real-world data while outlining limitations related to inter-reflections and polarized environment light.

Abstract

We present NeRSP, a Neural 3D reconstruction technique for Reflective surfaces with Sparse Polarized images. Reflective surface reconstruction is extremely challenging as specular reflections are view-dependent and thus violate the multiview consistency for multiview stereo. On the other hand, sparse image inputs, as a practical capture setting, commonly cause incomplete or distorted results due to the lack of correspondence matching. This paper jointly handles the challenges from sparse inputs and reflective surfaces by leveraging polarized images. We derive photometric and geometric cues from the polarimetric image formation model and multiview azimuth consistency, which jointly optimize the surface geometry modeled via implicit neural representation. Based on the experiments on our synthetic and real datasets, we achieve the state-of-the-art surface reconstruction results with only 6 views as input.

Figures (10)

• Figure 1: Shape recoveries of a reflective surface from 6 sparse polarized images capturing (top rows). Our NeRSP achieves a better shape reconstruction result compared to existing methods that either address sparse inputs (S-VolSDF wu2023s) or reflective reflectance (PANDORA dave2022pandora).
• Figure 2: Visualization of polarized images, derived AoP map, and Stokes vectors.
• Figure 3: Ambiguity of determining 3D postions under sparse views with geometric and photometric cues.
• Figure 4: Pipeline of our NeRSP.
• Figure 5: Capture setup and overview of our real-world multiview polarized image dataset RMVP3D.