MaskBlur: Spatial and Angular Data Augmentation for Light Field Image Super-Resolution

TL;DR

Extensive experiments demonstrate the efficacy of MaskBlur in significantly enhancing the performance of existing SR methods and it is extended to other LF image tasks such as denoising, deblurring, low-light enhancement, and real-world SR.

Abstract

Data augmentation (DA) is an effective approach for enhancing model performance with limited data, such as light field (LF) image super-resolution (SR). LF images inherently possess rich spatial and angular information. Nonetheless, there is a scarcity of DA methodologies explicitly tailored for LF images, and existing works tend to concentrate solely on either the spatial or angular domain. This paper proposes a novel spatial and angular DA strategy named MaskBlur for LF image SR by concurrently addressing spatial and angular aspects. MaskBlur consists of spatial blur and angular dropout two components. Spatial blur is governed by a spatial mask, which controls where pixels are blurred, i.e., pasting pixels between the low-resolution and high-resolution domains. The angular mask is responsible for angular dropout, i.e., selecting which views to perform the spatial blur operation. By doing so, MaskBlur enables the model to treat pixels differently in the spatial and angular domains when super-resolving LF images rather than blindly treating all pixels equally. Extensive experiments demonstrate the efficacy of MaskBlur in significantly enhancing the performance of existing SR methods. We further extend MaskBlur to other LF image tasks such as denoising, deblurring, low-light enhancement, and real-world SR. Code is publicly available at \url{https://github.com/chaowentao/MaskBlur}.

Figures (13)

• Figure 1: Comparison MaskBlur with CutBlur yoo2020rethinking and CutMIB xiao2023cutmib on various LF image SR methods, i.e., ATO jin2020light, , IINet liu2021intra, DistgSSR wang2022disentangling and EPIT liang2023learning. We compare the average PSNR (dB, $\uparrow$) on the HCIold wanner2013datasets dataset. Note that MaskBlur improves the values of PSNR by a large margin compared to other DA schemes on various LF image SR methods.
• Figure 2: Detailed comparisons of MaskBlur with CutBlur yoo2020rethinking and CutMIB xiao2023cutmib. MaskBlur consists of spatial blur and angular dropout, which are implemented through the spatial and angular masks, respectively.
• Figure 3: Analyzing CutBlur yoo2020rethinking, CutMIB xiao2023cutmib and MaskBlur from spatial and angular domain. The center SAI is in a 5 $\times$ 5 LF. The red rectangle denotes the area for the cutting and pasting operation, i.e., cut an LR patch and paste it to the original HR image. Macro-pixel image (MacPI) represents the pasted region in the angular domain. The spatial edge map is generated by using Sobel operation on the augmented HR LF image the angular error map is calculated between LR and augmented HR LF in the angular domain. MaskBlur can maintain spatial structure consistency and enhance angular domain information. Please zoom in for better visualization.
• Figure 4: Semantic illustration of MaskBlur. Here, an LF of size $U=V=3, H=W=3$ is used as a toy example. LR LF SAIs and HR LF SAIs are painted with green and blue background color. Different textures denote different views. We show the examples of $LR \rightarrow HR$ and $HR \rightarrow LR$. Better to view in color and texture.
• Figure 5: Qualitative comparison of the ATO jin2020light with and without MaskBlur when the network inputs the mask blurred image during the inference. Red boxes indicate the regions of the spatial mask. $\Delta$ is the absolute residual intensity map between the network output and the ground-truth HR image. MaskBlur successfully generates clear results in more blurred areas (red boxes) while the ATO generates unrealistic artifacts.