Towards Robust and Generalizable Lensless Imaging with Modular Learned Reconstruction

TL;DR

The paper tackles robustness and generalizability in lensless imaging by introducing a modular reconstruction framework that places a pre-processor before camera inversion and a post-processor after inversion. It combines physics-based forward models with learnable components, and demonstrates theoretical and empirical benefits of the pre-processor to mitigate model mismatch and noise amplification, across multiple mask types and datasets. A programmable DigiCam hardware platform and wave-based modeling enable broad generalization studies, including multi-mask training, PSF correction, and transfer learning, all open-sourced to accelerate adoption. The findings show improved image fidelity and robustness under varying SNRs and PSFs, with practical implications for scalable, low-cost, privacy-conscious lensless imaging in real-world settings.

Abstract

Lensless cameras disregard the conventional design that imaging should mimic the human eye. This is done by replacing the lens with a thin mask, and moving image formation to the digital post-processing. State-of-the-art lensless imaging techniques use learned approaches that combine physical modeling and neural networks. However, these approaches make simplifying modeling assumptions for ease of calibration and computation. Moreover, the generalizability of learned approaches to lensless measurements of new masks has not been studied. To this end, we utilize a modular learned reconstruction in which a key component is a pre-processor prior to image recovery. We theoretically demonstrate the pre-processor's necessity for standard image recovery techniques (Wiener filtering and iterative algorithms), and through extensive experiments show its effectiveness for multiple lensless imaging approaches and across datasets of different mask types (amplitude and phase). We also perform the first generalization benchmark across mask types to evaluate how well reconstructions trained with one system generalize to others. Our modular reconstruction enables us to use pre-trained components and transfer learning on new systems to cut down weeks of tedious measurements and training. As part of our work, we open-source four datasets, and software for measuring datasets and for training our modular reconstruction.

Figures (24)

• Figure 1: Modular lensless imaging pipeline. Pre- and post-processors and PSF correction are optional.
• Figure 2: DigiCam prototype and measurement setup.
• Figure 3: Point spread functions of datasets used in this work, where (meas) refers to a measured PSF and (sim) refers to simulated. \ref{['fig:multi1', 'fig:multi2', 'fig:multi3', 'fig:multi4']} are four of 100 simulated PSFs of the mask patterns used in measuring the DigiCam-Multi dataset.
• Figure 4: Visualization of improvement in image quality metrics when splitting the number of model parameters between pre- and post-processors, rather than only using a post-processor.
• Figure 5: Visual comparison of reconstructions on test set examples of datasets of different mask types (amplitude and phase). Models are trained on the corresponding training set of each dataset/system.