Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers

TL;DR

This work introduces the Panoramic Computational Imaging Engine (PCIE), a framework that integrates a minimalist PAL-based optical path (MPIP) with PSF-guided post-processing to achieve high-quality 360° panoramas. Central to PCIE is PART, a PSF-aware Transformer that leverages a novel PSF map alongside PSF-aware modules (PFM and PMAB) to recover aberration-heavy, low-resolution MPIP imagery for two tasks: Aberration Correction (AC) and Super-Resolution and Aberration Correction (SR\&AC). To support learning, PALHQ—a high-quality PAL image dataset—and synthetic wave-based simulations are used to train and evaluate models, with RealMPAL datasets for real-world benchmarks. The results demonstrate that PSF representation substantially boosts performance across synthetic and real data, enabling state-of-the-art restoration with minimal optical hardware. The work also provides practical insights into optical design, network architecture, data preparation, and training strategies for minimalist panoramic imaging, with PALHQ proving essential for robust model generalization. Overall, PCIE offers a viable path toward lightweight, wearable 360° imaging systems without sacrificing quality.

Abstract

High-quality panoramic images with a Field of View (FoV) of 360° are essential for contemporary panoramic computer vision tasks. However, conventional imaging systems come with sophisticated lens designs and heavy optical components. This disqualifies their usage in many mobile and wearable applications where thin and portable, minimalist imaging systems are desired. In this paper, we propose a Panoramic Computational Imaging Engine (PCIE) to achieve minimalist and high-quality panoramic imaging. With less than three spherical lenses, a Minimalist Panoramic Imaging Prototype (MPIP) is constructed based on the design of the Panoramic Annular Lens (PAL), but with low-quality imaging results due to aberrations and small image plane size. We propose two pipelines, i.e. Aberration Correction (AC) and Super-Resolution and Aberration Correction (SR&AC), to solve the image quality problems of MPIP, with imaging sensors of small and large pixel size, respectively. To leverage the prior information of the optical system, we propose a Point Spread Function (PSF) representation method to produce a PSF map as an additional modality. A PSF-aware Aberration-image Recovery Transformer (PART) is designed as a universal network for the two pipelines, in which the self-attention calculation and feature extraction are guided by the PSF map. We train PART on synthetic image pairs from simulation and put forward the PALHQ dataset to fill the gap of real-world high-quality PAL images for low-level vision. A comprehensive variety of experiments on synthetic and real-world benchmarks demonstrates the impressive imaging results of PCIE and the effectiveness of the PSF representation. We further deliver heuristic experimental findings for minimalist and high-quality panoramic imaging. Our dataset and code will be available at https://github.com/zju-jiangqi/PCIE-PART.

Figures (17)

• Figure 1: Illustration of the proposed MPIP and its key issue of low image quality, which is properly addressed with PSF-aware transformer: PART. (a) Minimalist Panoramic Imaging Prototype (MPIP); (b) Comparison between real products of conventional panoramic imaging systems and PAL-based MPIP; (c) Low-quality image captured by MPIP. (d) High-quality image recovered by PART. In this way, we realize minimalist and high-quality panoramic imaging with PSF-aware transformers.
• Figure 2: The proposed plug-and-play PSF-aware mechanism, PFM, consistently and significantly improves the performance of several baseline models in two pipelines. "+" means the model inserted the PFM in the same way as PART.
• Figure 3: Overview of the proposed Panoramic Computational Imaging Engine (PCIE) for minimalist and high-quality panoramic imaging. To achieve the goal of panoramic imaging with a minimalist system, the number of optical components and the radius of MPIP are designed to be small, which brings two key issues of low image quality: (1) aberration-induced blur due to lack of enough lenses for aberration correction and (2) low resolution caused by limited image plane size. We introduce the PART, which is trained on synthetic data pairs generated by imaging simulation, to recover the low-quality aberration image with the guidance of PSF information. "ISP" denotes Image Signal Processing.
• Figure 4: Two prototype samples of MPIP. Up: MPIP-P1, Down: MPIP-P2. (a) Optical path diagram. (b) Visualized PSF distributions. (c) The degraded checkerboard image patches of normalized FoVs $0.1$, $0.6$, and $0.9$ are captured by two MPIP samples. The minimalist optical design brings spatially-variant aberration-induced blur, especially for MPIP-P2 equipped with fewer lenses.
• Figure 5: Illustration of two pipelines for processing aberration-images of MPIP. (a) Comparison of image sensors with different pixel sizes. For a diffused spot through the optical system with a fixed size, more pixels of the sensor with smaller pixel sizes and higher resolutions are affected. (b) Raised two tasks based on sensors with different pixel sizes: Aberration Correction (AC) and Super-Resolution and Aberration Correction (SR$\&$AC). In summary, we target the recovery of a high-quality image from an aberration image of MPIP.