Spatial Polarization Multiplexing: Single-Shot Invisible Shape and Reflectance Recovery

TL;DR

This work introduces Spatial Polarization Multiplexing (SPM), a single-shot polarized-pattern method that jointly recovers an object's shape and reflectance while remaining invisible to the eye. By encoding AoLP per-pixel with a constrained de Bruijn sequence, SPM enables robust decoding, polarimetric diffuse–specular decomposition, and BRDF estimation from one image, including dynamic scenes through shifted patterns for adaptive resolution. The approach combines polarization theory (AoLP/DoLP, Stokes/Mueller calculus) with a tailored pattern design and dynamic programming-based decoding to achieve accurate geometry and radiometry without altering appearance. Experimental results on static and dynamic objects demonstrate precise depth, normal maps, polarimetric decomposition, relighting, and adaptive high-resolution sensing, highlighting the method's potential for unseen sensing and real-time 3D appearance capture.

Abstract

We propose spatial polarization multiplexing (SPM) for joint sensing of shape and reflectance of a static or dynamic deformable object, which is also invisible to the naked eye. Past structured-light methods are limited to shape acquisition and cannot recover reflectance as they alter scene appearance. Our key idea is to spatially multiplex a polarization pattern to encode the incident ray and also densely sample the reflected light. We derive a quantized polarized light pattern that can be robustly and uniquely decoded from the reflected Angle of Linear Polarization (AoLP) values. It also enables single-shot disentanglement of polarimetric diffuse and specular reflections for accurate BRDF estimation. We achieve this spatial polarization multiplexing (SPM) with a constrained de Bruijn sequence. We validate this novel invisible single-shot shape and reflectance method with real static and dynamic objects. The results demonstrate the effectiveness of SPM for accurate shape and BRDF measurement which opens new avenues of application for 3D sensing thanks to its invisibility and ability to jointly recover the radiometric properties.

Figures (19)

• Figure 1: Spatial Polarization Multiplexing (SPM) recovers the shape and reflectance of a target in a single shot with a novel polarimetric structured light pattern that enables simultaneous decoding for shape reconstruction and decomposition of polarimetric diffuse and specular reflections. SPM enables these joint sensing per-frame of deforming objects without altering the appearance (invisible to the naked eyes). Left: a plush toy pushed down with a finger. Right: a soft loaf expanding after compression.
• Figure 2: We design an SPM pattern with quantized stripes of AoLP values. Per-pixel polarization control enables robust decoding based on neighboring pixels and decomposition of captured polarimetric image into polarimetric diffuse and specular reflections for BRDF reconstruction.
• Figure 3: Pattern detection and decoding using quantized AoLP values. As shown in the inset, the observed AoLPs are corruptlll due to diffuse reflection. "Detected": the quantized AoLPs are incorrect where the corruption exceeds tolerance. "Decode": our decoding corrects these mislabelings.
• Figure 4: Example of our polarization pattern for $k_\mathrm{db}=6$, $n_\mathrm{db}=3$, and AoLP range $[0^\circ,80^\circ]$. We impose constraints on the de Bruijn sequence to prevent adjacent stripes with identical or near AoLP values for polarimetric decomposition and robust detection of the pattern.
• Figure 5: We smooth and shift the SPM pattern for high-resolution reconstruction of a static scene. We also change the AoLP assignment for each pattern to ensure sufficient variations of projected polarization states. The projected Stokes vectors are scaled and translated. The matching between the observed and projected Stokes vectors provides pixel-wise correspondences.