PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications

Abstract

Polarimetric imaging enables advanced vision applications such as normal estimation and de-reflection by capturing unique surface-material interactions. However, existing applications (alternatively called downstream tasks) rely on datasets constructed by naively regrouping raw measurements from division-of-focal-plane sensors, where pixels of the same polarization angle are extracted and aligned into sparse images without proper demosaicking. This reconstruction strategy results in suboptimal, incomplete targets that limit downstream performance. Moreover, current demosaicking methods are task-agnostic, optimizing only for photometric fidelity rather than utility in downstream tasks. Towards this end, we propose PolarAPP, the first framework to jointly optimize demosaicking and its downstream tasks. PolarAPP introduces a feature alignment mechanism that semantically aligns the representations of demosaicking and downstream networks via meta-learning, guiding the reconstruction to be task-aware. It further employs an equivalent imaging constraint for demosaicking training, enabling direct regression to physically meaningful outputs without relying on rearranged data. Finally, a task-refinement stage fine-tunes the task network using the stable demosaicking front-end to further enhance accuracy. Extensive experimental results demonstrate that PolarAPP outperforms existing methods in both demosaicking quality and downstream performance. Code is available upon acceptance.

Figures (19)

• Figure 1: (a) Comparison between two baselines and our joint PolarAPP, which directly performs polarization-based tasks from a single CPFA raw image. (b) The current SOTA pipeline, SfPUEL lyu2024sfpuel, suffers from artifacts in the reconstructed normal map due to noisy AoP estimation. (c) A naïve combination of the SOTA demosaicker PIDSR zhou2025pidsr with SfPUEL still fails to improve normal estimation. (d) In contrast, our PolarAPP yields sharper AoP and a more accurate normal map through customized joint learning with equivalent imaging transformation (EIT) and feature alignment.
• Figure 1: Metrics comparisons on demosaicking.
• Figure 2: The pipeline of PolarAPP. (a) Inference of PolarAPP. (b) Overview of the meta-learning-based iteration for joint training. (c) Inner update for updating demosaicking and task network under one gradient descent. (d) Outer update for updating the feature transform modules. (e) Joint demosaicking and task learning.
• Figure 3: Grad-CAM visual comparison of $\mathcal{L}_t$ and $\mathcal{L}_{fa}$ on SfP and DfP. It can be seen that $\mathcal{L}_{fa}$ yields more distributed, structure-aware responses for both $\mathcal{D}$ and $\mathcal{T}$.
• Figure 4: Learning demosaicking based on equivalent imaging transformation prior.