Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network

TL;DR

Restoring light-field images captured in extremely low-light conditions is challenging due to heavy noise and the need to preserve LF parallax. The authors propose DCUNet, an interpretable deep unfolding framework that alternates illumination-map estimation with enhancement, guided by the forward model $L_d = I \odot L_n + N$ and solved via half-quadratic splitting. Key contributions include a signal-dependent illumination map estimation scheme, a content-associated deep compensation (CDC) module to suppress noise and estimation bias, and a deep optimization step with a learned regularizer, all built upon a pseudo-explicit feature interaction backbone that exploits EPIs and ray convergence. Experiments on synthetic and real LF datasets show state-of-the-art performance in PSNR/SSIM/LPIPS and improved preservation of LF geometry, with publicly available code. This approach enables robust LF restoration under ultra-low-light conditions for downstream tasks in depth estimation and computational photography.

Abstract

This paper presents a novel and interpretable end-to-end learning framework, called the deep compensation unfolding network (DCUNet), for restoring light field (LF) images captured under low-light conditions. DCUNet is designed with a multi-stage architecture that mimics the optimization process of solving an inverse imaging problem in a data-driven fashion. The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result. Additionally, DCUNet includes a content-associated deep compensation module at each optimization stage to suppress noise and illumination map estimation errors. To properly mine and leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module that comprehensively exploits redundant information in LF images. The experimental results on both simulated and real datasets demonstrate the superiority of our DCUNet over state-of-the-art methods, both qualitatively and quantitatively. Moreover, DCUNet preserves the essential geometric structure of enhanced LF images much better. The code will be publicly available at https://github.com/lyuxianqiang/LFLL-DCU.

Figures (15)

• Figure 1: (a) Two key factors affecting brightness in the LF imaging process: incoming light intensity and ray scattering by the lenslet array. (b) An example illustrating the reduction in brightness in sub-aperture images as a consequence of scattered rays, as compared to the Lytro Illum camera preview.
• Figure 2: Flowchart of the proposed end-to-end learning-based framework for low-light LF image enhancement, namely DCUNet. DCUNet consists of three main modules: Illumination Map Estimation (IE), Content-associated Deep Compensation (CDC), and Deep Optimization (DO). The CDC module compensates for the solution bias of the enhanced intermediate results $\textbf{L}_n^{k-1}$ and estimated illumination map $\textbf{I}^{k}$. The DO module is designed to mimic the optimization process and iteratively refine the enhanced result.
• Figure 3: Demonstration of the utilization of redundant information by different strategies in an epipolar plane image (EPI) perspective. The orange blocks with different saturations represent corresponding pixels affected by severe noise, and the green bounding boxes show the scope of different operations.
• Figure 4: Illustration of (a) the pseudo-explicit feature interaction module and (b) the corresponding dense-skip connection.
• Figure 5: Detail architecture of the proposed pseudo-explicit feature (DPEF) interaction module. The input of the DPEF encompasses dense feature stack (spatial, angular, and epipolar plane image) information, with stack parameter $D\in \left \{ 1,..., L \right \}$, where $L$ represents the layer number. The kernel size is 3 and the stride is 1 for all the basic 4D-2D convolutions. To balance the quality of the low-light LF enhancement and computational expenses, we ultimately settled on 6 layers, and 32 channel numbers in our framework.