PIDSR: Complementary Polarized Image Demosaicing and Super-Resolution

TL;DR

PIDSR addresses the challenge of obtaining high-resolution, polarization-preserving imagery directly from CPFA raw data by jointly performing polarized image demosaicing and super-resolution. It introduces a two-stage recurrent pipeline—spatial-physical coherence reconstruction and polarization-aware resolution enhancement—coupled with a Stokes-aided network that injects physical polarization cues, formulated to maximize a posterior over polarimetric outputs. The approach yields state-of-the-art results on synthetic and real data, improving DoP and AoP accuracy and benefiting downstream tasks such as polarization-based reflection removal. This work enables more reliable, high-fidelity polarized imaging from CPFA sensors, with practical impact for polarization-guided vision applications.

Abstract

Polarization cameras can capture multiple polarized images with different polarizer angles in a single shot, bringing convenience to polarization-based downstream tasks. However, their direct outputs are color-polarization filter array (CPFA) raw images, requiring demosaicing to reconstruct full-resolution, full-color polarized images; unfortunately, this necessary step introduces artifacts that make polarization-related parameters such as the degree of polarization (DoP) and angle of polarization (AoP) prone to error. Besides, limited by the hardware design, the resolution of a polarization camera is often much lower than that of a conventional RGB camera. Existing polarized image demosaicing (PID) methods are limited in that they cannot enhance resolution, while polarized image super-resolution (PISR) methods, though designed to obtain high-resolution (HR) polarized images from the demosaicing results, tend to retain or even amplify errors in the DoP and AoP introduced by demosaicing artifacts. In this paper, we propose PIDSR, a joint framework that performs complementary Polarized Image Demosaicing and Super-Resolution, showing the ability to robustly obtain high-quality HR polarized images with more accurate DoP and AoP from a CPFA raw image in a direct manner. Experiments show our PIDSR not only achieves state-of-the-art performance on both synthetic and real data, but also facilitates downstream tasks.

Figures (6)

• Figure 1: Top: The concept of polarized image demosaicing (PID) and polarized image super-resolution (PISR). Mid: An example shows that the baseline (PID$\rightarrow$PISR) works in a sequential manner, where the AoPs calculated from the demosaicing results (produced by TCPDNet nguyen2022two) and the HR polarized images (produced by PSRNet hu2023polarized) suffer from severe artifacts. Bottom: An example shows that our PIDSR works in a complementary manner, where the calculated AoPs are more accurate. We choose $k=4$ here.
• Figure 2: (a) The average error rates of $\mathbf{p}$ and $\bm{\theta}$ are much larger than the one of $\mathbf{S}_0$ for PID methods (Polanalyser maeda2019polanalyser, IGRI2 morimatsu2021monochrome, and TCPDNet nguyen2022two). (b) The performance of PISR methods (PSRNet hu2023polarized and CPSRNet yu2023color) on both $\mathbf{p}$, $\bm{\theta}$, and $\mathbf{S}_0$ is much better when using mosaic-free ground truth polarized images compared with using demosaicing results. (c) The average error rates of both $\mathbf{p}$, $\bm{\theta}$, and $\mathbf{S}_0$ decrease as the resolution increases.
• Figure 3: A CPFA raw image $\mathbf{R}$ can be approximately converted to four half-resolution, full-color polarized image $\mathbf{R}_{\alpha_{1,2,3,4}}$.
• Figure 4: The workflow and network design of our PIDSR framework, consisting of two stages: spatial-physical coherence reconstructor $f(\cdot)$ and polarization-aware resolution enhancer $g(\cdot)$. Here we only illustrate the demosaicing workflow, and the SR one is in a repeated manner.
• Figure 5: Qualitative comparisons on both synthetic (the top group) and real data (the bottom group) of both demosaicing and 4$\times$ SR tasks.