LKFormer: Large Kernel Transformer for Infrared Image Super-Resolution

TL;DR

The paper addresses infrared image super-resolution by tackling the inefficiency of self-attention-based Transformers for 2D infrared data and the need to capture both local details and global context. It introduces LKFormer, which replaces standard self-attention with a Large Kernel Residual Attention (LKRA) built on depth-wise convolutions with multiple large kernels, enabling linear-complexity feature modeling. Complementing LKRA, the Gated-Pixel Feed-Forward Network (GPFN) adds a pixel-attention pathway to refine information flow for dense pixel prediction. Through extensive experiments on IR700, results-A, and ESPOL FIR, LKFormer achieves state-of-the-art PSNR/SSIM with fewer parameters, aided by ablations that validate the LKRA design and gating mechanism, and it provides practical implications for efficient infrared SR on resource-limited devices.

Abstract

Given the broad application of infrared technology across diverse fields, there is an increasing emphasis on investigating super-resolution techniques for infrared images within the realm of deep learning. Despite the impressive results of current Transformer-based methods in image super-resolution tasks, their reliance on the self-attentive mechanism intrinsic to the Transformer architecture results in images being treated as one-dimensional sequences, thereby neglecting their inherent two-dimensional structure. Moreover, infrared images exhibit a uniform pixel distribution and a limited gradient range, posing challenges for the model to capture effective feature information. Consequently, we suggest a potent Transformer model, termed Large Kernel Transformer (LKFormer), to address this issue. Specifically, we have designed a Large Kernel Residual Attention (LKRA) module with linear complexity. This mainly employs depth-wise convolution with large kernels to execute non-local feature modeling, thereby substituting the standard self-attentive layer. Additionally, we have devised a novel feed-forward network structure called Gated-Pixel Feed-Forward Network (GPFN) to augment the LKFormer's capacity to manage the information flow within the network. Comprehensive experimental results reveal that our method surpasses the most advanced techniques available, using fewer parameters and yielding considerably superior performance.The source code will be available at https://github.com/sad192/large-kernel-Transformer.

Figures (4)

• Figure 1: The architecture of the proposed LKFormer for infrared image SR. Here, RDB, RTB, and TL denote the residual depth-wise convolution block, residual Transformer block, and Transformer layer respectively. LKRA stands for large kernel residual attention block, which utilizes multiple convolutional layers of different sizes to extract features and then generates an attention map as a way to achieve encoding both local structural information and long-range dependencies of the input features. GPFN refers to a module called gated-pixel feed-forward network, which adds a branch of pixel attention to the classical feed-forward neural network as a way to control the forward flow of features. Additionally, LN and DWC3 represent the layer normalization and the $3\times3$ depth-wise convolution.
• Figure 2: Visual comparisons of LKFormer with other SR methods on IR700 dataset.
• Figure 3: LAM results of SwinIR, HAT, and LKFormer (Ours) in $\times4$ infrared image SR. When reconstructing the patches highlighted in red boxes, the red-marked areas represent informative pixels contributing to the reconstruction. Darker colors indicate a higher degree of contribution. The diffusion index (DI) reflects the range of pixels utilized in image reconstruction, with a higher DI showing a wider range and the involvement of more pixels. The results demonstrate that our method possesses a broader receptive domain, enabling a wider utilization of pixels for image reconstruction.
• Figure 4: The effect of PSNR measure of the different models with respect to training epochs for scaling factors of 2 and 4.