Representing Domain-Mixing Optical Degradation for Real-World Computational Aberration Correction via Vector Quantization

TL;DR

This paper delivers a novel insight into real-world CAC from the perspective of Unsupervised Domain Adaptation (UDA), and proposes a novel Quantized Domain-Mixing Representation (QDMR) framework as a potent solution to the issue.

Abstract

Relying on paired synthetic data, existing learning-based Computational Aberration Correction (CAC) methods are confronted with the intricate and multifaceted synthetic-to-real domain gap, which leads to suboptimal performance in real-world applications. In this paper, in contrast to improving the simulation pipeline, we deliver a novel insight into real-world CAC from the perspective of Unsupervised Domain Adaptation (UDA). By incorporating readily accessible unpaired real-world data into training, we formalize the Domain Adaptive CAC (DACAC) task, and then introduce a comprehensive Real-world aberrated images (Realab) dataset to benchmark it. The setup task presents a formidable challenge due to the intricacy of understanding the target optical degradation domain. To this intent, we propose a novel Quantized Domain-Mixing Representation (QDMR) framework as a potent solution to the issue. Centering around representing and quantizing the optical degradation which is consistent across different images, QDMR adapts the CAC model to the target domain from three key aspects: (1) reconstructing aberrated images of both domains by a VQGAN to learn a Domain-Mixing Codebook (DMC) characterizing the optical degradation; (2) modulating the deep features in CAC model with DMC to transfer the target domain knowledge; and (3) leveraging the trained VQGAN to generate pseudo target aberrated images from the source ones for convincing target domain supervision. Extensive experiments on both synthetic and real-world benchmarks reveal that the models with QDMR consistently surpass the competitive methods in mitigating the synthetic-to-real gap, which produces visually pleasant real-world CAC results with fewer artifacts. Codes and datasets are made publicly available at https://github.com/zju-jiangqi/QDMR.

Figures (11)

• Figure 3: Overview of the proposed QDMR. (a): To characterize the domain-mixing priors of optical degradation, we first pretrain a VQGAN to learn the DMC. (b): The pre-trained DMC is leveraged to guide the image restoration feature in the QDMR-Base model. The bottleneck module $B_{cac}$ can be any backbone for low-level vision tasks.
• Figure 4: Illustration of the proposed QDMR-UDA framework. (a): The VQGAN is exploited to transform the source images to the target domain, which generates pseudo paired target images for supervision. (b): Based on the s2t data flow and the QDMR-Base model, we develop the QDMR-UDA framework to further adapt the base model to the target domain through UDA training with a feature alignment strategy.
• Figure 5: The applied optical systems in Realab. First row: the optical paths of the applied MOS. Bottom two rows: Imaging results of the two optical systems with different aberration behaviors.
• Figure 6: Illustration of the simulated synthetic-to-real gap in Real-Sim. The simulated domain gap makes the degradation distribution of Real-Sim deviate from that of Syn.
• Figure 7: Illustration of the affine-based fusion module.