Glossy Object Reconstruction with Cost-effective Polarized Acquisition

TL;DR

The paper tackles glossy object reconstruction by leveraging a cost-effective acquisition setup that captures multi-view polarization images with an unknown polarizer angle. It introduces a neural implicit surface framework (VolSDF-style) with decomposed diffuse and specular radiance nets, guided by a differentiable polarization rendering layer that uses the polarimetric BRDF to estimate the outgoing Stokes vector $\mathbf{s}^\text{out}$ and render at $\phi_\text{pol}$. Optimization of the loss $\mathcal{L} = \mathcal{L}_\text{rgb} + \mathcal{L}_\text{mask} + 0.1\mathcal{L}_\text{eikonal}$ yields high-fidelity geometry and radiance decomposition, demonstrated on public and real-world datasets with robustness to polarizer-angle variations. The approach reduces hardware costs and enables accurate reconstruction and novel-view synthesis, with potential applications on consumer devices such as smartphones and IoTs.

Abstract

The challenge of image-based 3D reconstruction for glossy objects lies in separating diffuse and specular components on glossy surfaces from captured images, a task complicated by the ambiguity in discerning lighting conditions and material properties using RGB data alone. While state-of-the-art methods rely on tailored and/or high-end equipment for data acquisition, which can be cumbersome and time-consuming, this work introduces a scalable polarization-aided approach that employs cost-effective acquisition tools. By attaching a linear polarizer to readily available RGB cameras, multi-view polarization images can be captured without the need for advance calibration or precise measurements of the polarizer angle, substantially reducing system construction costs. The proposed approach represents polarimetric BRDF, Stokes vectors, and polarization states of object surfaces as neural implicit fields. These fields, combined with the polarizer angle, are retrieved by optimizing the rendering loss of input polarized images. By leveraging fundamental physical principles for the implicit representation of polarization rendering, our method demonstrates superiority over existing techniques through experiments in public datasets and real captured images on both reconstruction and novel view synthesis.

Figures (10)

• Figure 1: We build a cost-effective data acquisition system for capturing multi-view polarization images, where a linear polarizer is mounted in front of the off-the-shelf RGB camera and a single image per-view with unknown angle of the polarizer is captured, which eliminates the need for precise alignment. For objects with a hybrid of ceramics (tummy) and metal (feet), we can still nicely recover the specular components and estimate the polarimetric states, directly leading to high-fidelity geometry.
• Figure 2: Overview of neural glossy object reconstruction with polarization cues. Our method consists of three main steps (1--3): data acquisition, neural radiance field-based representation, and polarization rendering. This work employs neural rendering techniques in conjunction with the fundamental principles of polarization to generate a polarized image. These coupled modules allow for acquiring only one single polarization image at each viewing angle and then recover geometry and material properties through the optimization of rendering loss. Components marked with upward diagonal strips, such as $\mathbf{DiffuseNet}$ and $\mathbf{SpecularNet}$, are optimized during training, while those with grid checker patterns are calculated using corresponding equations.
• Figure 3: Qualitative results of captured datasets. For each scenario, the top row shows the input reference image, ground-truth mesh (obtained by painting and scanning), and corresponding normals; the bottom row demonstrates our resolved results, including the rendered image and extracted mesh.
• Figure 4: Qualitative comparison with SOTA methods. Our approach excels in reconstructing intricate features such as beard and tail segments, due to the advantage of the polarization information.
• Figure 5: Comparison of reflectance separation and surface normals with baselines on rendered Bust model. Note that, although PANDORA outputs sharp results, our method is also able to produce comparable results, because overall we use fewer constraints and need to solve for more unknowns.