SPIDeRS: Structured Polarization for Invisible Depth and Reflectance Sensing

TL;DR

SPIDeRS tackles invisible capture of 3D shape and surface reflectance by projecting per-pixel AoLP polarization patterns and decoding them with an RGB-polarimetric camera. The core idea is to use a TN-LC-based polarization projector to modulate AoLP at each pixel, enabling simultaneous recovery of depth, per-pixel surface normals, and BRDF without altering the surface appearance, and even enabling relighting. The method combines a physically grounded pBRDF model (FMBRDF) with a robust AoLP extraction pipeline that accounts for ambient and diffuse components, achieving depth maps comparable to traditional methods while preserving texture. This invisible sensing capability holds promise for real-world xR, robotics, and HCI applications, where maintaining natural appearance during sensing is crucial.

Abstract

Can we capture shape and reflectance in stealth? Such capability would be valuable for many application domains in vision, xR, robotics, and HCI. We introduce structured polarization for invisible depth and reflectance sensing (SPIDeRS), the first depth and reflectance sensing method using patterns of polarized light. The key idea is to modulate the angle of linear polarization (AoLP) of projected light at each pixel. The use of polarization makes it invisible and lets us recover not only depth but also directly surface normals and even reflectance. We implement SPIDeRS with a liquid crystal spatial light modulator (SLM) and a polarimetric camera. We derive a novel method for robustly extracting the projected structured polarization pattern from the polarimetric object appearance. We evaluate the effectiveness of SPIDeRS by applying it to a number of real-world objects. The results show that our method successfully reconstructs object shapes of various materials and is robust to diffuse reflection and ambient light. We also demonstrate relighting using recovered surface normals and reflectance. We believe SPIDeRS opens a new avenue of polarization use in visual sensing.

Figures (11)

• Figure 1: We propose structured polarization, a novel invisible 3D sensing method using polarized light with per-pixel modulation of the angle of linear polarization. The method enables completely stealth measurement of 3D shape, surface normals, and reflectance.
• Figure 2: DoLP and AoLP rotation through TN liquid crystal. The angle $\phi$ is the AoLP of the incident light. The TN liquid crystal enables us to control the AoLP arbitrarily in the range of $90^\circ$.
• Figure 3: AoLP pattern extraction from polarimetric reflection on the object. Diffuse reflection and ambient light alter the AoLP throw in the reflected polarimetric object appearance and cause decoding errors (left). Our extraction method robustly extracts the true projected pattern (right).
• Figure 4: Calibration results of DoLP and AoLP when the polarizing filter before the SLM is horizontal (Filter 1) and rotated by $45^\circ$ (Filter 2). We can continuously rotate the AoLP in the range of $[0^\circ, 90^\circ]$ by changing the pixel value. The DoLP is large enough to be detected for the horizontal filter.
• Figure 5: Polarization projector-camera system implementing SPIDeRS. Our polarization projector consists of an LED telecentric illuminator with a polarizer, SLM, and convex lens.