High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution

TL;DR

This work tackles the computational burden of image super-resolution by introducing a training-free adaptive masking strategy that prioritizes high-frequency regions. A high-frequency map, produced via Gaussian blur subtraction, is binarized using K-means to form a mask that guides pixel- or token-level processing in CNNs and Transformers, enabling inference-time pruning without retraining. The approach integrates seamlessly with both CNNs and Vision Transformers through unfolding for CNNs and window-based pruning for Transformers, achieving 24–43% FLOPs reductions while maintaining PSNR on par with full-computation baselines. The method is robust to unseen degradations and offers a scalable path to faster SR on diverse architectures, with source code available for reproducibility.

Abstract

The primary challenge in accelerating image super-resolution lies in reducing computation while maintaining performance and adaptability. Motivated by the observation that high-frequency regions (e.g., edges and textures) are most critical for reconstruction, we propose a training-free adaptive masking module for acceleration that dynamically focuses computation on these challenging areas. Specifically, our method first extracts high-frequency components via Gaussian blur subtraction and adaptively generates binary masks using K-means clustering to identify regions requiring intensive processing. Our method can be easily integrated with both CNNs and Transformers. For CNN-based architectures, we replace standard $3 \times 3$ convolutions with an unfold operation followed by $1 \times 1$ convolutions, enabling pixel-wise sparse computation guided by the mask. For Transformer-based models, we partition the mask into non-overlapping windows and selectively process tokens based on their average values. During inference, unnecessary pixels or windows are pruned, significantly reducing computation. Moreover, our method supports dilation-based mask adjustment to control the processing scope without retraining, and is robust to unseen degradations (e.g., noise, compression). Extensive experiments on benchmarks demonstrate that our method reduces FLOPs by 24--43% for state-of-the-art models (e.g., CARN, SwinIR) while achieving comparable or better quantitative metrics. The source code is available at https://github.com/shangwei5/AMSR

Figures (6)

• Figure 1: Visualization of the $\times 4$ SR results of the image "1400" from Test4K. Compared to other methods, our method employs high-frequency information to adaptively identify regions requiring network processing at the pixel level, thereby achieving comparable PSNR performance while significantly reducing computation. Masks of different colors indicate the varying levels of complexity with which different methods process distinct regions of the image.
• Figure 2: Overview of key differences in various architectures. (a) For CNN-based methods, we utilize the unfold operation to expand features to achieve a receptive field equivalent to that of a $3\times 3$ convolution, facilitating subsequent spatially invariant $1\times 1$ convolution operations. During training, to ensure backward propagation, we apply $\bm m$ to exclude regions that do not require processing. In the testing phase, we sample pixels that require processing based on $\bm m$, thereby reducing computation. (b) We compute pruning decisions $\bm{m}_{win}$ based on the mask corresponding to each token. During training, all tokens are fed into the module, while during testing, only the necessary tokens are retained to complete acceleration. The terms "AVG" and "$\sigma$" refer to the average pooling operation and the threshold of the average pooling operation, respectively.
• Figure 3: Visual comparison of our methods with backbone networks, and SOTA dynamic SISR method with $\times 4$ super-resolution. These examples are image “1258” (above) from Test2K, image “1399” (middle) from Test4K, and image “1405” (below) from Test8K respectively. Our method achieves comparable performance and lower computation except for FSRCNN.
• Figure 4: Visual comparison of our method with the backbone network. These examples are image “1300” from Test2K.
• Figure 5: Illustration of the $\times 4$ SR results of the image "1244" from Test2K under different degradations. The first row presents the results when the degradations are aligned with those used during training kong2021classsr. The second row displays the results employing a more extensive and complex set of random degradations zhang2021designing.