Computational Imaging for Machine Perception: Transferring Semantic Segmentation beyond Aberrations

TL;DR

This work addresses semantic segmentation under optical aberrations in Minimalist Optical Systems (MOS) by introducing a wave-based imaging simulation and Virtual Prototype Lens (VPL) benchmark to enable unsupervised domain adaptation for SSOA. It presents Computational Imaging Assisted Domain Adaptation (CIADA), which couples a Bidirectional Teacher and correlation-based knowledge distillation to transfer Computational Imaging priors into the segmentation task without increasing inference cost. Experiments show transformer-based segmenters are more robust to MOS aberrations, and CIADA consistently yields superior mIoU gains across aberration distributions compared to baselines, effectively bridging the gap between CI and downstream machine perception. The work provides Cityscapes-ab and KITTI-360-ab datasets and a project page, advancing MOS-enabled semantic perception for mobile and wearable platforms.

Abstract

Semantic scene understanding with Minimalist Optical Systems (MOS) in mobile and wearable applications remains a challenge due to the corrupted imaging quality induced by optical aberrations. However, previous works only focus on improving the subjective imaging quality through the Computational Imaging (CI) technique, ignoring the feasibility of advancing semantic segmentation. In this paper, we pioneer the investigation of Semantic Segmentation under Optical Aberrations (SSOA) with MOS. To benchmark SSOA, we construct Virtual Prototype Lens (VPL) groups through optical simulation, generating Cityscapes-ab and KITTI-360-ab datasets under different behaviors and levels of aberrations. We look into SSOA via an unsupervised domain adaptation perspective to address the scarcity of labeled aberration data in real-world scenarios. Further, we propose Computational Imaging Assisted Domain Adaptation (CIADA) to leverage prior knowledge of CI for robust performance in SSOA. Based on our benchmark, we conduct experiments on the robustness of classical segmenters against aberrations. In addition, extensive evaluations of possible solutions to SSOA reveal that CIADA achieves superior performance under all aberration distributions, bridging the gap between computational imaging and downstream applications for MOS. The project page is at https://github.com/zju-jiangqi/CIADA.

Figures (14)

• Figure 1: We pioneer scene understanding with Minimalist Optical Systems (MOS). The proposed Computational Imaging Assisted Domain Adaptation (CIADA) can achieve more accurate segmentation on the aberration image of MOS, compared to other possible solutions, e.g. Src-Only (SegFormer xie2021segformer) and CI&Seg (applying NAFNet chen2022simple before SegFormer xie2021segformer), without the extra computational overhead. The result is comparable to conventional lenses with sophisticated lens designs. The inference time of each model and the total inference time of each possible solution are shown in the "()".
• Figure 2: Relative performance (Rel., the normalized mIoU after being divided by that of the score of the Oracle) of possible solutions to Semantic Segmentation under Optical Aberrations (SSOA). Following michaelis2019benchmarkinghummer2023vltseg, we also apply the robustness metric of relative performance under corruption (rPC) over the four aberration levels for a comprehensive evaluation, which is shown in the bottom left corner of the figure ($\%$). The aberration level is the defined incremental level for different severity of aberrations, which will be depicted in Sec. \ref{['sec:benchmark']}. The relative results reveal that CIADA outperforms other solutions at all levels.
• Figure 3: We summarize two aberrations behaviors for MOS: (a) Common Simple Lens (CSL), revealing spatial-variant degradation. (b) Hybrid Refractive Diffractive Lens (HRDL), revealing spatial-uniform degradation.
• Figure 4: The simulation pipeline for constructing virtual prototype lens. The optical system is considered a black box with phase function $\Phi$. We can simulate the imaging result of any MOS through the pipeline.
• Figure 5: A novel benchmark for Semantic Segmentation under Optical Aberrations (SSOA). We construct Virtual Prototype Lens (VPL) groups with different behaviors and levels of aberrations to map clear images into simulated imaging results. C1-C4: aberration levels for CSL, H1-H4: aberration levels for HRDL. A hybrid set of C1/H1 to C4/H4 composes the level C5/H5, for comprehensive training and evaluation of SSOA solutions.