Learning to Deblur Polarized Images

TL;DR

This work tackles motion blur in polarization images captured by DoFP cameras, which complicates accurate computation of $p$ and $\\bm{\\theta}$. It proposes a polarization-aware deblurring pipeline that splits the task into deblurring an unpolarized image and then restoring polarized channels under polarization constraints, implemented as a two-stage network: an unpolarized image estimator and a polarized image reconstructor. A tailored loss combining content and Stokes priors, along with a synthetic polarization dataset and training strategy, enables effective learning. Empirical results on synthetic and real data show state-of-the-art performance and notable improvements for polarization-based vision tasks such as dehazing and reflection removal.

Abstract

A polarization camera can capture four linear polarized images with different polarizer angles in a single shot, which is useful in polarization-based vision applications since the degree of linear polarization (DoLP) and the angle of linear polarization (AoLP) can be directly computed from the captured polarized images. However, since the on-chip micro-polarizers block part of the light so that the sensor often requires a longer exposure time, the captured polarized images are prone to motion blur caused by camera shakes, leading to noticeable degradation in the computed DoLP and AoLP. Deblurring methods for conventional images often show degraded performance when handling the polarized images since they only focus on deblurring without considering the polarization constraints. In this paper, we propose a polarized image deblurring pipeline to solve the problem in a polarization-aware manner by adopting a divide-and-conquer strategy to explicitly decompose the problem into two less ill-posed sub-problems, and design a two-stage neural network to handle the two sub-problems respectively. Experimental results show that our method achieves state-of-the-art performance on both synthetic and real-world images, and can improve the performance of polarization-based vision applications such as image dehazing and reflection removal.

Figures (5)

• Figure 1: Architecture of the neural network that implements the proposed polarized image deblurring pipeline. It consists of two stages: an unpolarized image estimator that uses $\mathbf{S}^*_{1,2}$ (the blurry counterparts of the Stokes parameters $\mathbf{S}_{1,2}$) as priors to estimate the sharp unpolarized image $\mathbf{I}_{\text{guide}}$, and a polarized image reconstructor that reconstructs $\mathbf{I}_{\alpha_{1,2,3,4}}$ under the guidance of $\mathbf{I}_{\text{guide}}$. All images are normalized for visualization.
• Figure 2: Left: Two of the Stokes parameters ($\mathbf{S}_{1,2}$) have similar "appearance" to the gradient distribution $\mathbf{G}$ of the corresponding unpolarized image (please zoom-in for better details). Middle: LPIPS zhang2018perceptual scores between the blurry/sharp variable pairs, including $\mathbf{S}_{1,2}$ and $\mathbf{G}$. Right: Semantic distance between $\mathbf{I}_{\alpha_{1,2,3,4}}$ and $\mathbf{I}$, $\mathbf{I}_{\alpha_{1,2,3,4}}$ and $\mathbf{B}_{\alpha_{1,2,3,4}}$.
• Figure 3: Qualitative comparisons on synthetic data among our method, IPLNet hu2020iplnet, ColorPolarNet xu2022colorpolarnet, PLIE zhou2023polarization, MIMO cho2021rethinking, XYDeblur ji2022xydeblur, STDAN zhang2022spatio, and MISCFilter liu2024motion. The DoLP $\mathbf{p}$ and AoLP $\bm{\theta}$ are visualized using color maps after normalizing and averaging the RGB channels (as done in hu2020iplnetzhou2023polarization). Please zoom-in for better details.
• Figure 4: Qualitative comparisons on real data among our method, IPLNet hu2020iplnet, ColorPolarNet xu2022colorpolarnet, PLIE zhou2023polarization, MIMO cho2021rethinking, XYDeblur ji2022xydeblur, STDAN zhang2022spatio, and MISCFilter liu2024motion. The DoLP $\mathbf{p}$ and AoLP $\bm{\theta}$ are visualized using color maps after normalizing and averaging the RGB channels (as done in hu2020iplnetzhou2023polarization). Please zoom-in for better details.
• Figure 5: Results of polarization-based vision applications (including (a) image dehazing zhou2021learning and (b) reflection removal lyu2019reflectionlyu2022physics) without deblurring and deblurred by our method and the compared methods (IPLNet hu2020iplnet, ColorPolarNet xu2022colorpolarnet, PLIE zhou2023polarization, MIMO cho2021rethinking, XYDeblur ji2022xydeblur, STDAN zhang2022spatio, and MISCFilter liu2024motion). Please zoom-in for better details.