Arbitrary Volumetric Refocusing of Dense and Sparse Light Fields

TL;DR

The paper tackles the problem of refocusing multiple arbitrary regions within both dense and sparse light fields, addressing the limitation of prior methods that could not selectively refocus regions within the same depth range. It proposes an end-to-end pipeline that uses pixel-dependent shifts in a shift-and-sum framework, combined with a depth-guided alpha mask for dense LFs and an exhaustive alpha search for cross-shaped sparse LFs. To counteract ghosting artifacts in sparse data, it introduces a U-Net-based image restoration network trained with dense-LF ground truth, yielding high-fidelity refocused results even with limited data. The approach demonstrates efficient processing and high similarity metrics across multiple datasets, enabling near-real-time performance for sparse LFs and offering practical benefits for LF photography and cinematography.

Abstract

A four-dimensional light field (LF) captures both textural and geometrical information of a scene in contrast to a two-dimensional image that captures only the textural information of a scene. Post-capture refocusing is an exciting application of LFs enabled by the geometric information captured. Previously proposed LF refocusing methods are mostly limited to the refocusing of single planar or volumetric region of a scene corresponding to a depth range and cannot simultaneously generate in-focus and out-of-focus regions having the same depth range. In this paper, we propose an end-to-end pipeline to simultaneously refocus multiple arbitrary planar or volumetric regions of a dense or a sparse LF. We employ pixel-dependent shifts with the typical shift-and-sum method to refocus an LF. The pixel-dependent shifts enables to refocus each pixel of an LF independently. For sparse LFs, the shift-and-sum method introduces ghosting artifacts due to the spatial undersampling. We employ a deep learning model based on U-Net architecture to almost completely eliminate the ghosting artifacts. The experimental results obtained with several LF datasets confirm the effectiveness of the proposed method. In particular, sparse LFs refocused with the proposed method archive structural similarity index higher than 0.9 despite having only 20% of data compared to dense LFs.

Figures (7)

• Figure 1: Refocusing of "Bush" LF; (a) single planar refocus ng2005light; (b) two volumetric regions multivolume; (c) proposed arbitrary volumetric refocusing using a dense LF; (d) proposed arbitrary volumetric refocusing using a sparse LF having cross-shaped SAIs. For (c) and (d), narrow-depth and wide-depth regions are shown in boxes with green- and red-colored outlines, respectively.
• Figure 2: Different types of LF cameras; (a) LF video camera array LFVideoCamera, (b) Lytro Illum dense LF camera from Lytro, Inc, (c) EPIModule Sparse LF camera from EPIImaging LLC.
• Figure 3: The two-plane parameterization of an LF. The $uv$ plane is called the camera plane, and the $xy$ plane is called the image plane. The distance between the two planes are $F$, and the scene at a distance $F$ is in focus while the rest are out of focus. The image plane is moved artificially to the $x'y'$ plane (called refocused plane) to refocus to a new depth $\alpha F$.
• Figure 4: The flow charts of (a) dense LF refocusing algorithm and (b) sparse LF refocusing algorithm. The user can interactively select an ROI or multiple ROIs on middle SAI which are needed to be refocused. For dense LFs, first, the depth map is estimated. Then generate the $\mathcal{M}_{\alpha}(n_x,n_y)$. Finally refocus each region with different $\alpha$ value separately and concatenate together to get the final refocused image. For sparse LFs, first suitable $\alpha$ value search is done for each ROI. Then create the $\mathcal{M}_{\alpha}(n_x,n_y)$ and refocus each region with different $\alpha$ value and concatenate. Finally, send through the image restoration CNN to remove the aliasing artifacts and get the final refocused image.
• Figure 5: Sections of $\mathcal{M}_{\alpha}(n_x,n_y)$ of multi arbitrary-volume refocused LF shown in \ref{['fig:dense_multi_refocused_img_slices']} as a heatmap.