RGB-to-Polarization Estimation: A New Task and Benchmark Study

TL;DR

This work introduces RGB-to-polarization estimation, a sensor-free task that predicts polarization information from standard RGB images using the Stokes parameter framework. It establishes the first benchmark on the Jeon et al. RGB–polarization dataset to compare restoration-based and generation-based models, revealing that pre-trained representations (e.g., MAE, Stable Diffusion with LoRA) and strong restoration backbones yield the best quantitative and qualitative polarization estimates. The study analyzes the strengths and limitations of direct reconstruction versus generative synthesis, and discusses practical directions such as incorporating physical constraints and self-supervised learning to improve robustness. By quantifying polarization estimation from RGB inputs and providing a standardized evaluation protocol, the work aims to democratize polarization imaging and guide future sensor-free polarization research.

Abstract

Polarization images provide rich physical information that is fundamentally absent from standard RGB images, benefiting a wide range of computer vision applications such as reflection separation and material classification. However, the acquisition of polarization images typically requires additional optical components, which increases both the cost and the complexity of the applications. To bridge this gap, we introduce a new task: RGB-to-polarization image estimation, which aims to infer polarization information directly from RGB images. In this work, we establish the first comprehensive benchmark for this task by leveraging existing polarization datasets and evaluating a diverse set of state-of-the-art deep learning models, including both restoration-oriented and generative architectures. Through extensive quantitative and qualitative analysis, our benchmark not only establishes the current performance ceiling of RGB-to-polarization estimation, but also systematically reveals the respective strengths and limitations of different model families -- such as direct reconstruction versus generative synthesis, and task-specific training versus large-scale pre-training. In addition, we provide some potential directions for future research on polarization estimation. This benchmark is intended to serve as a foundational resource to facilitate the design and evaluation of future methods for polarization estimation from standard RGB inputs.

Figures (30)

• Figure 1: Comparison between sensor-based methods and our polarization estimation approach. Conventional methods rely on physical acquisition systems (e.g., polarization cameras or rotating polarizers), whereas our method leverages RGB inputs and neural networks to estimate polarization information without requiring dedicated hardware. The predicted polarization images can be readily applied to a variety of downstream tasks.
• Figure 2: (a) The Poincaré sphere serves as a geometric representation for describing all possible states of polarization using the Stokes parameters. (b–d) Polarization describes how the electric field of a light wave oscillates within the plane perpendicular to the direction of propagation. (b) Linear polarizations at 0° and 90°. (c) Linear polarizations at $\pm 45^\circ$ angles. (d) Right- and left-handed circular polarizations.
• Figure 3: Qualitative comparison of estimated polarization components from RGB input. Results are shown for Uformer wang2022uformer, MAE he2022masked, DiT peebles2023scalable, and Img2ImgTurbo parmar2024one.
• Figure 4: Qualitative comparison of estimated polarization components from RGB input. Results are shown for Uformer wang2022uformer, MAE he2022masked, DiT peebles2023scalable, and Img2ImgTurbo parmar2024one.
• Figure 5: Qualitative comparison of predicted DoLP and AoLP maps, derived from Stokes components estimated from RGB input. Ground-truth maps are provided for reference, along with results from Uformer wang2022uformer, MAE he2022masked, DiT peebles2023scalable, and Img2ImgTurbo parmar2024one.