Efficient Attention-Sharing Information Distillation Transformer for Lightweight Single Image Super-Resolution

TL;DR

This work tackles the efficiency of Transformer-based single image super-resolution (SISR) by introducing ASID, a lightweight architecture that couples information distillation with attention-sharing to mitigate the high cost of self-attention. The model uses Information Distillation Blocks (IDBs) comprising Local Modules, Spatial and Channel Attention Modules, and a channel-split mechanism, with attention matrices shared across blocks to reduce computations. ASID achieves competitive PSNR/SSIM on standard benchmarks with only about 300K parameters, outperforming many lightweight CNN-based SR methods and approaching Transformer-based methods at a fraction of the cost. The design enables deeper self-attention stacks without prohibitive compute, making practical high-quality SISR feasible on resource-constrained devices, and code is provided on the project page.

Abstract

Transformer-based Super-Resolution (SR) methods have demonstrated superior performance compared to convolutional neural network (CNN)-based SR approaches due to their capability to capture long-range dependencies. However, their high computational complexity necessitates the development of lightweight approaches for practical use. To address this challenge, we propose the Attention-Sharing Information Distillation (ASID) network, a lightweight SR network that integrates attention-sharing and an information distillation structure specifically designed for Transformer-based SR methods. We modify the information distillation scheme, originally designed for efficient CNN operations, to reduce the computational load of stacked self-attention layers, effectively addressing the efficiency bottleneck. Additionally, we introduce attention-sharing across blocks to further minimize the computational cost of self-attention operations. By combining these strategies, ASID achieves competitive performance with existing SR methods while requiring only around 300K parameters - significantly fewer than existing CNN-based and Transformer-based SR models. Furthermore, ASID outperforms state-of-the-art SR methods when the number of parameters is matched, demonstrating its efficiency and effectiveness. The code and supplementary material are available on the project page.

Figures (7)

• Figure 1: Visualized comparison of PSNR and the number of parameters on the Urban100 ($\times2$) dataset. Our ASID is compared with state-of-the-art lightweight SR methods. Green markers represent CNN-based methods, while yellow markers denote Transformer-based methods.
• Figure 2: Visualized overall structure of ASID. ASID mainly consists of a convolutional layer for shallow feature extraction, a series of IDBs, and an upsampling module that reconstructs features into an SR image. Blue arrows indicate the attention-sharing mechanism.
• Figure 3: Visualized structure of Information Distillation Blocks (IDBs) and the attention-sharing mechanism. Blue arrows represent the attention-sharing mechanism. PW-CONV denotes pixel-wise convolution.
• Figure 4: Visualized structure of the Local Module (LM), Spatial Attention Module (SAM), and Channel Attention Module (CAM). The blue arrow represents the attention-sharing mechanism in SAM. By employing the attention-sharing technique, subsequent SAMs bypass the calculation of the spatial attention matrix, which typically accounts for a significant portion of the computational load in self-attention layers. Meso-level self-attention computes attention matrices among pixels within the same partition, whereas global-level self-attention involves pixels from different partitions wang2023omni. Utilizing both methods effectively mitigates the limited receptive field issue associated with window-based self-attention. All feedforward networks are omitted for simplicity in visualization.
• Figure 5: Qualitative Comparison of previous CNN-based and Transformer-based SR methods on the Urban100 ($\times4$) dataset. Note that ASID accurately restores images while using an extremely small number of model parameters.