CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution

TL;DR

CATANet tackles the inefficiency of global self-attention in image super-resolution by introducing a Content-Aware Token Aggregation (CATA) mechanism that shares token centers across all image tokens and updates them only during training. It combines intra-group self-attention (IASA) over content-similar token subgroups with inter-group cross-attention (IRCA) to enable refined long-range and global interactions, while retaining local processing via Local-Region Self-Attention (LRSA) and a ConvFNN. The approach achieves state-of-the-art performance among lightweight SR methods, surpassing SPIN by up to 0.60 dB PSNR and roughly halving inference time, demonstrating strong practical impact for resource-constrained deployment. The results indicate that content-aware token aggregation, coupled with targeted attention mechanisms, can efficiently capture long-range dependencies without the heavy cost of full global attention.

Abstract

Transformer-based methods have demonstrated impressive performance in low-level visual tasks such as Image Super-Resolution (SR). However, its computational complexity grows quadratically with the spatial resolution. A series of works attempt to alleviate this problem by dividing Low-Resolution images into local windows, axial stripes, or dilated windows. SR typically leverages the redundancy of images for reconstruction, and this redundancy appears not only in local regions but also in long-range regions. However, these methods limit attention computation to content-agnostic local regions, limiting directly the ability of attention to capture long-range dependency. To address these issues, we propose a lightweight Content-Aware Token Aggregation Network (CATANet). Specifically, we propose an efficient Content-Aware Token Aggregation module for aggregating long-range content-similar tokens, which shares token centers across all image tokens and updates them only during the training phase. Then we utilize intra-group self-attention to enable long-range information interaction. Moreover, we design an inter-group cross-attention to further enhance global information interaction. The experimental results show that, compared with the state-of-the-art cluster-based method SPIN, our method achieves superior performance, with a maximum PSNR improvement of 0.33dB and nearly double the inference speed.

Figures (6)

• Figure 1: Performance and model complexity comparison on Urban100 dataset for upscaling factor ×2.
• Figure 2: The overall architecture of CATANet and the structure of Token Aggregation Block and Local-Region Self-Attenion.
• Figure 3: (a) A simple illustration for obtaining the initial token centers. (b) Visualization of sub-grouping. The dashed boxes of the same color indicate the same group (left) or subgroup (right). (c) Each subgroup's $\mathbf{Q}_j$ to attend to the $\mathbf{K}_j$ of two consecutive subgroups, where the same color denotes the same group $\mathcal{G}_j$
• Figure 4: Visualization of token grouping results of TAB. (a) is the input image. The white part of each binarized image from (b) - (e) represents a single token group $\mathcal{G}_j$.
• Figure 5: LAM Comparison: the variant without TAB (w/o TAB), the full CATANet (w/ TAB), RCAN and SwinIR-light for $\times$4 SR.