GRFormer: Grouped Residual Self-Attention for Lightweight Single Image Super-Resolution

TL;DR

This work tackles the efficiency-accuracy trade-off in transformer-based single image super-resolution by introducing GRFormer, a lightweight SR model built on Grouped Residual Self-Attention (GRSA). GRSA combines a Grouped Residual Linear (GRL) layer and Exponential-Space Relative Position Bias (ES-RPB) to dramatically reduce parameter counts and computations while boosting SR performance across $ imes$2, $ imes$3, and $ imes$4 scales. The authors provide thorough ablations showing the individual and combined contributions of GRL and ES-RPB, achieving state-of-the-art results among lightweight SR models, with notable gains on Urban100 and Manga109 and favorable running-time characteristics. The proposed GRSA framework offers a practical pathway for efficient, high-quality transformer-based SR and can be readily integrated into window-based SR architectures.

Abstract

Previous works have shown that reducing parameter overhead and computations for transformer-based single image super-resolution (SISR) models (e.g., SwinIR) usually leads to a reduction of performance. In this paper, we present GRFormer, an efficient and lightweight method, which not only reduces the parameter overhead and computations, but also greatly improves performance. The core of GRFormer is Grouped Residual Self-Attention (GRSA), which is specifically oriented towards two fundamental components. Firstly, it introduces a novel grouped residual layer (GRL) to replace the Query, Key, Value (QKV) linear layer in self-attention, aimed at efficiently reducing parameter overhead, computations, and performance loss at the same time. Secondly, it integrates a compact Exponential-Space Relative Position Bias (ES-RPB) as a substitute for the original relative position bias to improve the ability to represent position information while further minimizing the parameter count. Extensive experimental results demonstrate that GRFormer outperforms state-of-the-art transformer-based methods for $\times$2, $\times$3 and $\times$4 SISR tasks, notably outperforming SOTA by a maximum PSNR of 0.23dB when trained on the DIV2K dataset, while reducing the number of parameter and MACs by about \textbf{60\%} and \textbf{49\% } in only self-attention module respectively. We hope that our simple and effective method that can easily applied to SR models based on window-division self-attention can serve as a useful tool for further research in image super-resolution. The code is available at \url{https://github.com/sisrformer/GRFormer}.

Figures (4)

• Figure 1: (a) shows the comparisons of self-attention of recent transformer-based SR models in terms of multiply-accumulate operations (MACs) and parameters. (b) shows SISR comparisons of recent SR models ($\times$4) in terms of PSNR on Urban100, network parameters. Our model (GRFormer) outperforms the SOTA model ($\times$4) by 0.19dB in PSNR score while having comparably low network parameters and MACs.
• Figure 2: Comparison between RPB and ES-RPB. The subfigure (a) and (b) showcase the relative position bias (RPB) from the SwinIR model and an GRFomer model where RPB are replaced to ES-RPB, respectively. The subfigure at $i_{th}$ row and $j_{th}$ column corresponds to the relative position bias of $i_{th}$ GRSAB Group and $j_{th}$ GRSAB in the network. These figures specifically highlight the horizontal evolution of the relative position bias values. The x-axis extends from 0 to 62, while the y-axis corresponds to the data taken at the 7th point on the x-axis from an RPB matrix of size 15$\times$63.
• Figure 3: Network architecture of the proposed GRFormer. It mainly consists of a shallow feature extraction module, several grouped residual self-attention block groups (GRSAB Group) to learn deep feature mapping in an efficient and effective way, and a high-resolution image reconstruction module.
• Figure 4: Qualitative comparison with recent state-of-the-art lightweight image SR methods on the ×4 SR task.