HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution

TL;DR

This work tackles the efficiency and receptive-field limitations of transformer-based image super-resolution (SR). It introduces HiT-SR, a hierarchical transformer framework that uses expanding hierarchical windows to capture multi-scale features and long-range dependencies, coupled with a spatial-channel correlation (SCC) module that achieves linear complexity with window size. By converting popular SR transformers (SwinIR-Light, SwinIR-NG, SRFormer-Light) into HiT-SR variants, the approach delivers improved SR quality with fewer parameters and faster inference, achieving about a 7x speed-up over baselines while preserving or enhancing accuracy. The combination of hierarchical window design and SCC enables scalable, efficient utilization of large receptive fields, establishing new state-of-the-art results across standard SR benchmarks and scales.

Abstract

Transformers have exhibited promising performance in computer vision tasks including image super-resolution (SR). However, popular transformer-based SR methods often employ window self-attention with quadratic computational complexity to window sizes, resulting in fixed small windows with limited receptive fields. In this paper, we present a general strategy to convert transformer-based SR networks to hierarchical transformers (HiT-SR), boosting SR performance with multi-scale features while maintaining an efficient design. Specifically, we first replace the commonly used fixed small windows with expanding hierarchical windows to aggregate features at different scales and establish long-range dependencies. Considering the intensive computation required for large windows, we further design a spatial-channel correlation method with linear complexity to window sizes, efficiently gathering spatial and channel information from hierarchical windows. Extensive experiments verify the effectiveness and efficiency of our HiT-SR, and our improved versions of SwinIR-Light, SwinIR-NG, and SRFormer-Light yield state-of-the-art SR results with fewer parameters, FLOPs, and faster speeds ($\sim7\times$).

Figures (8)

• Figure 1: Comparisons of the popular efficient SR transformers, i.e., SwinIR-Light (SIR) liang2021swinir, SwinIR-NG (SNG) choi2023ngram, and SRFormer-Light (SRF) Zhou_2023srformer, and the corresponding HiT-SR versions, i.e., HiT-SIR, HiT-SNG, and HiT-SRF. The complexity metrics are calculated under $\times 2$ upscaling on an A100 GPU, with the output size set to $720 \times 1280$.
• Figure 2: Typical framework for transformer-based SR methods, where the block-level and layer-level improvements made by our HiT-SR are colored red. SA, FFN, and LN indicate self-attention, feed-forward network, and layer normalization, respectively.
• Figure 3: Block-Level design in HiT-SR. Hierarchical windows are applied to different transformer layers (TLs) to aggregate features with expanding receptive fields.
• Figure 4: Layer-Level design in HiT-SR composed of dual feature extraction (DFE), spatial and channel self-correlation (S-SC and C-SC). DFE is designed to extract features from spatial and channel domains. S-SC and C-SC are proposed to efficiently aggregate hierarchical information with linear computational complexity to window sizes.
• Figure 5: Convergence comparison of SwinIR-Light liang2021swinir and our improved version HiT-SIR on Urban100 ($\times 2$) and Manga109 ($\times 2$) datasets under the same training settings.