Towards Physics-informed Cyclic Adversarial Multi-PSF Lensless Imaging

TL;DR

This work tackles the fragility of GAN-based lensless imaging to PSF variations by introducing a physics-informed cyclic adversarial framework that jointly uses a forward model and dual discriminators to learn PSF-aware reconstructions. It proposes two PSF-aware generator architectures (TU-Net and Y-Net) that fuse sparse PSF features via sparse convolutions or PSF subdivision, enabling robust multi-PSF lensless imaging without retraining. A derived DiffuserCam/Mirflickr-based dataset with PSF shuffling augments training, and the forward model is efficiently integrated into the training loop to enforce physics-consistent cycle losses. Quantitative and qualitative results show the approach achieves PSF-agnostic performance comparable to camera-specific methods in single-PSF cases and superior robustness across multiple PSFs, highlighting substantial gains in generalization and practical applicability for compact computational cameras.

Abstract

Lensless imaging has emerged as a promising field within inverse imaging, offering compact, cost-effective solutions with the potential to revolutionize the computational camera market. By circumventing traditional optical components like lenses and mirrors, novel approaches like mask-based lensless imaging eliminate the need for conventional hardware. However, advancements in lensless image reconstruction, particularly those leveraging Generative Adversarial Networks (GANs), are hindered by the reliance on data-driven training processes, resulting in network specificity to the Point Spread Function (PSF) of the imaging system. This necessitates a complete retraining for minor PSF changes, limiting adaptability and generalizability across diverse imaging scenarios. In this paper, we introduce a novel approach to multi-PSF lensless imaging, employing a dual discriminator cyclic adversarial framework. We propose a unique generator architecture with a sparse convolutional PSF-aware auxiliary branch, coupled with a forward model integrated into the training loop to facilitate physics-informed learning to handle the substantial domain gap between lensless and lensed images. Comprehensive performance evaluation and ablation studies underscore the effectiveness of our model, offering robust and adaptable lensless image reconstruction capabilities. Our method achieves comparable performance to existing PSF-agnostic generative methods for single PSF cases and demonstrates resilience to PSF changes without the need for retraining.

Figures (14)

• Figure 1: An overview of the cyclic adversarial framework. $G_{y,k}$ refers to the generator that takes lensless image $y$ from domain $Y$ and PSF $k$ from domain $K$, as its inputs. $F_{x,k}$ refers to the forward model that takes the lensed image $x$ from domain $X$ and the PSF $k$ from domain $K$, as its inputs.
• Figure 2: Our dual discriminator generative adversarial reconstruction methodology for multi-PSF lensless imaging. Here $G$ is the generator, $F$ is the forward model, and $D_{1}, D_{2}$ are the two discriminators.
• Figure 3: The detailed architectures of the Y-generator and the TU-generator. In the Y-generator configuration, the CNN encoder and the decoder have skip connections, and the auxiliary sparse encoder merges at the bottleneck giving a distinct Y-shaped appearance. In the TU-generator configuration, the T-structure results due to the CNN encoder-decoder forming the upper branch and the auxiliary sparse encoder forming the stem.
• Figure 4: The complete dual-discriminator physics-informed cyclic adversarial training pipeline for multi-PSF lensless imaging. Generator $G$ takes the lensless image $y$ and the PSF $k$ as its input. The forward model $F$ takes the lensed image $x$ and PSF $k$ as its input. The left half of the figure illustrates the cyclic domain translation from lensed $X$ to lensless $Y$ and back. Similarly, the right half illustrates the cyclic domain translation from lensless $Y$ to lensed $X$ and back.
• Figure 5: Reconstructions corresponding to multiple-PSFs compared against Rego et al. rego2021robust. The highlighted portions inside the green square indicate that our method can noticeably outperform the existing multi-PSF method, especially in terms of image sharpness. The Y-Net framework was used to obtain these reconstructions.