Beyond Subspace Isolation: Many-to-Many Transformer for Light Field Image Super-resolution

TL;DR

The paper tackles subspace isolation in light-field image super-resolution by introducing a Many-to-Many Transformer (M2MT) that aggregates angular information in the spatial subspace before self-attention, enabling access to all SAIs and non-local spatial-angular modeling. Built atop this, the M2MT-Net combines M2MT in the spatial domain with a vanilla Angular Transformer to achieve state-of-the-art LFSR performance with favorable memory and compute efficiency. Through comprehensive experiments, it demonstrates superior PSNR/SSIM across multiple LF datasets, sharper qualitative details, improved angular consistency via depth maps, and strong interpretability via Local Attribution Maps. Ablation studies confirm the effectiveness of the spatial M2MT component and optimal correlation-channel settings, while analyses on efficiency highlight practical scalability for real-world LF applications.

Abstract

The effective extraction of spatial-angular features plays a crucial role in light field image super-resolution (LFSR) tasks, and the introduction of convolution and Transformers leads to significant improvement in this area. Nevertheless, due to the large 4D data volume of light field images, many existing methods opted to decompose the data into a number of lower-dimensional subspaces and perform Transformers in each sub-space individually. As a side effect, these methods inadvertently restrict the self-attention mechanisms to a One-to-One scheme accessing only a limited subset of LF data, explicitly preventing comprehensive optimization on all spatial and angular cues. In this paper, we identify this limitation as subspace isolation and introduce a novel Many-to-Many Transformer (M2MT) to address it. M2MT aggregates angular information in the spatial subspace before performing the self-attention mechanism. It enables complete access to all information across all sub-aperture images (SAIs) in a light field image. Consequently, M2MT is enabled to comprehensively capture long-range correlation dependencies. With M2MT as the foundational component, we develop a simple yet effective M2MT network for LFSR. Our experimental results demonstrate that M2MT achieves state-of-the-art performance across various public datasets, and it offers a favorable balance between model performance and efficiency, yielding higher-quality LFSR results with substantially lower demand for memory and computation. We further conduct in-depth analysis using local attribution maps (LAM) to obtain visual interpretability, and the results validate that M2MT is empowered with a truly non-local context in both spatial and angular subspaces to mitigate subspace isolation and acquire effective spatial-angular representation.

Figures (7)

• Figure 1: Super-resolution results and local attribute maps (LAM) of the proposed M2MT-Net against EPIT on the Perforated_metal_3 sample.
• Figure 2: Illustration of accessible data in LF tensors used by existing LF Transformers under the One-to-One scheme and our proposed Many-to-Many Transformer. For the LF tensors, each color represents a SAI.
• Figure 3: Illustration of M2MT-Net and its components. (a) depicts the overview of M2MT. (b) and (c) illustrate the details of a M2MT Transformer and an angular Transformer. These two components constitute a Correlation Block in (d). $\bigoplus$ represents the addition operation of a residual connection.
• Figure 4: Visualization of selected samples in the $4\times$ task. In each sample, the following result is provided for each compared method: the SAI, the zoom-in views from the blue and red boxes, the PSNR/SSIM of the red box, the Local Attribution Map (LAM) of the red box and its Diffusion Index (DI). The best and second-best PSNR/SSIM are in bold and underlined. The angular location indicator is given below the HR.
• Figure 5: Visualization of SAI-wise PSNR to demonstrate the distribution of $4\times$ LFSR performance. The compared samples are the same with Fig. \ref{['fig:Qual']}.