NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics

TL;DR

This work proposes NeISF++, an inverse rendering pipeline that supports conductors and dielectrics and proposes a novel geometry initialization method using DoLP images that surpasses the existing polarized inverse rendering methods for geometry and material decomposition as well as downstream tasks like relighting.

Abstract

Recent inverse rendering methods have greatly improved shape, material, and illumination reconstruction by utilizing polarization cues. However, existing methods only support dielectrics, ignoring conductors that are found everywhere in life. Since conductors and dielectrics have different reflection properties, using previous conductor methods will lead to obvious errors. In addition, conductors are glossy, which may cause strong specular reflection and is hard to reconstruct. To solve the above issues, we propose NeISF++, an inverse rendering pipeline that supports conductors and dielectrics. The key ingredient for our proposal is a general pBRDF that describes both conductors and dielectrics. As for the strong specular reflection problem, we propose a novel geometry initialization method using DoLP images. This physical cue is invariant to intensities and thus robust to strong specular reflections. Experimental results on our synthetic and real datasets show that our method surpasses the existing polarized inverse rendering methods for geometry and material decomposition as well as downstream tasks like relighting.

Figures (8)

• Figure 1: Comparison of the polarized inverse rendering methods. Since NeISF li2024neisf does not model conductors correctly, its geometry and material reconstruction is poor. Our method estimates the complex refractive index of conductors and re-renders the image via a physically-based pBRDF. Therefore, the reconstructed material and geometry are accurate, and the relighting result is glossy. We normalize the real and imaginary parts of the complex refractive index $\bm{ior}$, and visualize them separately.
• Figure 2: Cosine values of phase delay (upper) and reflection coefficients (bottom) of a typical conductor (gold at 633nm) and dielectric (refractive index equals 1.5). "R_s" and "R_p" are the perpendicular and parallel components of reflectance, "avg" denotes their average value.
• Figure 3: Geometry initialization pipeline using both intensity and DoLP images. Intensity on the conductor area suffers from strong specular reflections, while DoLP images are less affected by strong specular reflections. The initialized signed distance fields $f_\text{sdf}$ will continue to be trained in the joint optimization stage.
• Figure 4: Overview of the joint optimization stage.
• Figure 5: Surface normal results of synthetic data. Mean angular errors are on the top. Our method shows a better reconstruction quality than NeRO liu2023nero and PANDORA dave2022pandora. NeISF li2024neisf failed because of the wrong material model and the poor geometry initialization of VolSDF yariv2021volume. Our geometry initialization method VolSDF-DoLP shows a better reconstruction quality.