Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors

LI Yang, WU Ruizheng, LI Jiyong, CHEN Ying-cong

TL;DR

Comparisons prove this method outperforms existing neural 3D reconstruction methods in reflective scenes by a large margin and proposes a reweighting strategy in the optimization process to alleviate the noise issue of polarization priors.

Abstract

Learning surfaces from neural radiance field (NeRF) became a rising topic in Multi-View Stereo (MVS). Recent Signed Distance Function (SDF)-based methods demonstrated their ability to reconstruct accurate 3D shapes of Lambertian scenes. However, their results on reflective scenes are unsatisfactory due to the entanglement of specular radiance and complicated geometry. To address the challenges, we propose a Gaussian-based representation of normals in SDF fields. Supervised by polarization priors, this representation guides the learning of geometry behind the specular reflection and captures more details than existing methods. Moreover, we propose a reweighting strategy in the optimization process to alleviate the noise issue of polarization priors. To validate the effectiveness of our design, we capture polarimetric information, and ground truth meshes in additional reflective scenes with various geometry. We also evaluated our framework on the PANDORA dataset. Comparisons prove our method outperforms existing neural 3D reconstruction methods in reflective scenes by a large margin.

GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors

TL;DR

Comparisons prove this method outperforms existing neural 3D reconstruction methods in reflective scenes by a large margin and proposes a reweighting strategy in the optimization process to alleviate the noise issue of polarization priors.

Abstract

Learning surfaces from neural radiance field (NeRF) became a rising topic in Multi-View Stereo (MVS). Recent Signed Distance Function (SDF)-based methods demonstrated their ability to reconstruct accurate 3D shapes of Lambertian scenes. However, their results on reflective scenes are unsatisfactory due to the entanglement of specular radiance and complicated geometry. To address the challenges, we propose a Gaussian-based representation of normals in SDF fields. Supervised by polarization priors, this representation guides the learning of geometry behind the specular reflection and captures more details than existing methods. Moreover, we propose a reweighting strategy in the optimization process to alleviate the noise issue of polarization priors. To validate the effectiveness of our design, we capture polarimetric information, and ground truth meshes in additional reflective scenes with various geometry. We also evaluated our framework on the PANDORA dataset. Comparisons prove our method outperforms existing neural 3D reconstruction methods in reflective scenes by a large margin.
Paper Structure (18 sections, 9 equations, 5 figures, 2 tables)

This paper contains 18 sections, 9 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-view 3D Reconstruction
  4. BRDF for Reflective Objects Reconstruction
  5. Multi-view 3D Reconstruction with Polarization
  6. Gaussians in 3D Scene Representation
  7. Methods
  8. Preliminary of Polarization
  9. Gaussian Guided Polarimetric Neural 3D Reconstruction Pipeline
  10. Learning Surface by Volume Rendering
  11. Gaussian Splatting of Normals
  12. Optimization with Reweighted Polarization Priors
  13. Experiments
  14. PolRef Dataset
  15. Experimental Settings
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Visualization of Gaussians of normals in Neural Reconstruction pipelines. 2D Gaussians can be rendered from 3D Gaussians of learned normals.
  • Figure 2: Illustration of polarization shift in specular reflection. The right figure is a detailed description of the geometry relation between AoP and the surface normal. $\psi$ is the azimuth angle. $\varphi$ is the AoP, which is the angle from the positive x-axis to the polarized direction.
  • Figure 3: Illustration of our method.
  • Figure 4: Visualization of Reweighted AoP Priors. Red boxes bound specular reflection dominant regions, and the blue boxes bound diffuse ones. (d) is the AoP map reweighted by DoP. Saturation in (e) indicates the degree of anisotropy, and color represents the direction of the singular vector of 2D Gaussians' covariance. A few 2D Gaussians are drawn as ellipses for intuition.
  • Figure 5: Visual comparison of our method and state-of-the-art methods.