Single-pixel imaging via data-driven and deep image prior dual networks

TL;DR

The paper addresses the challenge of high-quality reconstruction in single-pixel imaging under sub-sampling by combining data-driven networks (DD-Net) and deep image priors (DIP-Net) into a dual-network iterative optimization framework (SPI-DNIO). It introduces gradient-rich residual blocks (RGI) and a two-step training protocol that jointly optimizes projection patterns with physical constraints and then refines with a second network, achieving high fidelity with fewer iterations. Experimental and simulation results show superior SSIM and PSNR at 25% sampling, strong generalization across diverse datasets (LFW, DIV2K, Set14, LSDIR), and robustness to noise in both indoor and outdoor settings. The approach enhances practical SPI performance by leveraging complementary strengths of data priors and deep priors, with notable improvements in reconstruction quality, speed, and versatility for real-world imaging tasks.

Abstract

Single-pixel imaging(SPI),especially when integrated with deep neural networks like deep image prior networks (DIP-Net) or data-driven networks (DD-Net), has gained considerable attention for its capability to generate high-quality reconstructed images, even in the presence of sub-sampling conditions. However, DIP-Net often requires thousands of iterations to achieve high-quality image reconstruction, and DD-Net performs optimally only when the target closely resembles the features present in its training set. To overcome these limitations, we propose a dual-network iterative optimization (SPI-DNIO) framework that combines the strengths of both DD-Net and DIP-Net. It has been demonstrated that this approach can recover high-quality images with fewer iteration steps. Furthermore, to address the challenge of SPI inputs having less effective information at low sampling rates, we have designed a residual block enriched with gradient information, which can convey details to deeper layers, thereby enhancing the deep network's learning capabilities. We have applied these techniques to both indoor experiments with active lighting and outdoor long-range experiments with passive lighting. Our experimental results confirm the exceptional reconstruction capabilities and generalization performance of the SPI-DNIO framework.

Figures (9)

• Figure 1: The specific steps of the SPI-DNIO. (A) The process of network training and optimizing patterns. (B) The Iterative optimization for the image reconstruction. Imaging results of the LFW training dataset (Randomly select 10000 images from the LFW dataset) for 64 $\times$ 64 resolution images at 25% sampling ratio. The face images were taken from LFW34
• Figure 2: The framework of the RGI-RNet.
• Figure 3: (A) The imaging results of the reconstruction networks including SPI-DNIO, DIP-Net, and the PT-Net for 64$\times$64 images at a 25% sampling ratio. (B) Evaluation metrics for three reconstruction models and their corresponding error bars which are calculated through repeated experiments on the same device. The iteration step of SPI-DNIO and DIP-Net is 400. (C) The loss is calculated by Eq. (\ref{['eq:6']}) and the SSIM between the ground truth and the reconstruction along with the number of iteration steps from 0 to 400 for the target 'Abdullah_Gul_0015' in the LFW34 validation dataset.
• Figure 4: Robustness comparisons of different SPI reconstruction methods under various dSNR levels. The method naming explanations are as follows. PT represents the first frozen RGI-RNet in SPI-DNIO, which loads the PT-weight, while NPT stands for the random initialization of the first frozen network. ES refers to adopting the ES strategy during iterative optimization while NES means not employing the ES strategy. The image size is 64 $\times$ 64 and the sampling ratio is set to 25%. The "Face" and the "Pyramid" object images are selected from the LFW34 and DIV2K validation datasets, respectively. The iteration step of the SPI-DNIO is 400.
• Figure 5: Comparison of different SPI reconstruction approaches at different sampling rates. The target objects are selected from diverse datasets including LFW, SET14, DIV2K, and LSDIR. The RGI-RNet-s is a scaling down of the RGI-RNet, only decreasing the number of ResBlock to 9 and the structure of Unet is consistent with the reference 33. The image size is $64 \times 64$. The iteration step for SPI-DNIO and DIP-Net is 400. The networks utilized are indicated in parentheses. The face images were taken from LFW34