Probabilistic-based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising

TL;DR

A probabilistic-based feature embedding (PFE) is proposed, which learns a featureembedding architecture by assembling various low-dimensional convolution patterns in a probability space for fully capturing spatial-angular information.

Abstract

The high-dimensional nature of the 4-D light field (LF) poses great challenges in achieving efficient and effective feature embedding, that severely impacts the performance of downstream tasks. To tackle this crucial issue, in contrast to existing methods with empirically-designed architectures, we propose a probabilistic-based feature embedding (PFE), which learns a feature embedding architecture by assembling various low-dimensional convolution patterns in a probability space for fully capturing spatial-angular information. Building upon the proposed PFE, we then leverage the intrinsic linear imaging model of the coded aperture camera to construct a cycle-consistent 4-D LF reconstruction network from coded measurements. Moreover, we incorporate PFE into an iterative optimization framework for 4-D LF denoising. Our extensive experiments demonstrate the significant superiority of our methods on both real-world and synthetic 4-D LF images, both quantitatively and qualitatively, when compared with state-of-the-art methods. The source code will be publicly available at https://github.com/lyuxianqiang/LFCA-CR-NET.

Figures (18)

• Figure 1: (a) Illustration of probabilistic-based feature embedding. The learned network is derived from the MAP estimation in a probability space using a template network that comprises multiple layers of basic convolution of LF slices, including spatial, angular, vertical EPI, and horizontal EPI slices. Building upon this module, a cycle-consistent framework (b) has been devised for compressive imaging, which involves several iterative stages of projection and reconstruction. During the projection phase, the coded aperture imaging process is simulated. Additionally, we propose an iterative denoising framework (c) comprising multiple iterative noise suppression modules.
• Figure 2: Illustration of network-level feature aggregation and feature embedding unit used in our network. (a,c) The network with all potential paths, namely the template network. (b,d) One possible architecture from the posterior distribution $\mathcal{Q}(\mathcal{W}| \mathbb{D})$.
• Figure 3: Illustration of the proposed framework named CR-Net for reconstructing 4-D LF images from 2-D measurements by a coded-aperture camera. CR-Net consists of a "CMs Simulation" procedure and the proposed cycle-consistent reconstruction network.
• Figure 4: Illustration of the process of the proposed 4-D LF denoising method, namely DN-Net.
• Figure 5: Illustration of imaging model when the occlusion occurs. (a) The imaging process influenced by the occluder. (b) Corresponding angular patch by fixed spatial positions.