Recursive Generalization Transformer for Image Super-Resolution

TL;DR

This work tackles image super-resolution by enabling efficient global spatial modeling with a Transformer variant. It introduces RG-SA, which uses a recursive generalization module to produce compact representative maps and cross-attention to exchange global information, along with channel scaling to reduce redundancy. To balance local details, RG-SA is combined with local self-attention via Hybrid Adaptive Integration (HAI), which adaptively fuses multi-level features outside each block. Empirical results show that RGT achieves state-of-the-art PSNR/SSIM across standard SR benchmarks with favorable computational costs, and ablations validate the effectiveness of RG-SA and HAI.

Abstract

Transformer architectures have exhibited remarkable performance in image super-resolution (SR). Since the quadratic computational complexity of the self-attention (SA) in Transformer, existing methods tend to adopt SA in a local region to reduce overheads. However, the local design restricts the global context exploitation, which is crucial for accurate image reconstruction. In this work, we propose the Recursive Generalization Transformer (RGT) for image SR, which can capture global spatial information and is suitable for high-resolution images. Specifically, we propose the recursive-generalization self-attention (RG-SA). It recursively aggregates input features into representative feature maps, and then utilizes cross-attention to extract global information. Meanwhile, the channel dimensions of attention matrices (query, key, and value) are further scaled to mitigate the redundancy in the channel domain. Furthermore, we combine the RG-SA with local self-attention to enhance the exploitation of the global context, and propose the hybrid adaptive integration (HAI) for module integration. The HAI allows the direct and effective fusion between features at different levels (local or global). Extensive experiments demonstrate that our RGT outperforms recent state-of-the-art methods quantitatively and qualitatively. Code and pre-trained models are available at https://github.com/zhengchen1999/RGT.

Figures (6)

• Figure 1: The architecture of the Recursive Generalization Transformer (RGT). The local self-attention (L-SA) blocks, and recursive-generalization self-attention (RG-SA) blocks are alternately arranged. $\alpha_{l}$ is a learnable adaptor in the hybrid adaptive integration (HAI) of the $l^{th}$ block.
• Figure 2: The illustration of the recursive-generalization self-attention (RG-SA). The RG-SA first generates the representative feature maps with constant size ($h$$\times$$w$), through the recursive generalization module (RGM). Then, the cross-attention between input features and representative maps is performed to capture global information. $\otimes$ denotes matrix multiplication.
• Figure 3: Visualization of the features in RGT. a) Block input. (b) Block output before connection. (c) Vanilla skip connection result: directly add input to output. (d) HAI result: adjust input with $\alpha$ before adding to output. Please zoom in for a better visualization.
• Figure 4: CKA similarity between all pairs of Transformer blocks in RGT-S without or with HAI.
• Figure 5: The value ($|\alpha| / C$, $C$$=$$180$) distribution of $\alpha$ of RGT-S.