Fooling Polarization-based Vision using Locally Controllable Polarizing Projection

TL;DR

This work reveals a vulnerability in polarization-based vision by demonstrating physically realizable adversarial attacks that operate through locally controllable polarizing projection. By adapting a one-chip LCD projector (removing the front polarizer), the authors enable pixel-wise control over polarization without affecting intensity, enabling imperceptible perturbations to polarization cues. They formulate whitebox attacks against polarization-dependent tasks such as glass segmentation and shape-from-polarization (SfP) by optimizing a grid-based perturbation, leveraging an Expectation Over Transformation (EOT) strategy and a tailored adversarial loss that manipulates polarization-induced predictions. Experimental results in both digital simulations and real-world setups show significant degradation of polarization-based predictions while maintaining human-imperceptibility, highlighting new security risks and the need for defenses in polarization-aware AI systems.

Abstract

Polarization is a fundamental property of light that encodes abundant information regarding surface shape, material, illumination and viewing geometry. The computer vision community has witnessed a blossom of polarization-based vision applications, such as reflection removal, shape-from-polarization, transparent object segmentation and color constancy, partially due to the emergence of single-chip mono/color polarization sensors that make polarization data acquisition easier than ever. However, is polarization-based vision vulnerable to adversarial attacks? If so, is that possible to realize these adversarial attacks in the physical world, without being perceived by human eyes? In this paper, we warn the community of the vulnerability of polarization-based vision, which can be more serious than RGB-based vision. By adapting a commercial LCD projector, we achieve locally controllable polarizing projection, which is successfully utilized to fool state-of-the-art polarization-based vision algorithms for glass segmentation and color constancy. Compared with existing physical attacks on RGB-based vision, which always suffer from the trade-off between attack efficacy and eye conceivability, the adversarial attackers based on polarizing projection are contact-free and visually imperceptible, since naked human eyes can rarely perceive the difference of viciously manipulated polarizing light and ordinary illumination. This poses unprecedented risks on polarization-based vision, both in the monochromatic and trichromatic domain, for which due attentions should be paid and counter measures be considered.

Figures (8)

• Figure 1: (a) Single-chip color polarization sensor can capture trichromatic image, angle of linear polarization (AoLP), and degree of linear polarization (DoLP) within one shot. (b) Our proposed physical attackers are based on polarizing projection, which is naturally conceivable to human eyes, thus can bypass the trade-off between attack efficacy and eye conceivability in fooling RGB-based vision.
• Figure 2: (a) The principle of intensity adjustment in one-chip LCD projector with a liquid crystal panel sandwiched by two perpendicular linear polarizers. (b) The polarization direction of the light beam in each liquid crystal cell can be individually controlled, without affecting its intensity. (c) The range of controllable polarization direction.
• Figure 3: (a) The typical structure of a one-chip LCD projector, in which light emitted by LED lamp will go through a liquid crystal panel, mirror, and projection lens. (b) A linear polarizer is attached to the front side of the LCD panel. We tear it off to make our polarizing projection. (c)(d) The projection of a normal LCD projector and our adapted projector. The arrows' direction and length represent light polarizing direction and intensity, respectively. A normal projector emits out colorful light of constant polarization direction, while our adapted polarizing projector emits light with constant intensity but different polarizing angles. Note that, for naked eyes and ordinary RGB cameras, the projected light is completely uniform, even if their polarization directions are totally different. The projected image can be observed by eyes with the assistance of a linear polarizer on the screen.
• Figure 4: The Illustration of intensity, AoLP and DoLP images from our digital world simulation alongside their counterparts captured in the physical world. In the digital scenario with Expectation Over Transformation (EOT), the images are augmented through the addition of Gaussian noises, Gaussian blurring, and scaling of background intensity, all of which are randomly sampled to simulate real world degradations.
• Figure 5: Illustration of $\mathcal{L}_{BCE}$, and IoU during the optimization process.