From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution

TL;DR

The paper addresses the high computational cost of Transformer-based self-attention in image super-resolution by introducing individualized, content-aware attention through the Individualized Exploratory Attention (IEA) mechanism. IEA progressively expands attention candidates while sparsifying low-similarity connections, enabling tokens to adaptively attend to diverse, long-range sources without symmetric grouping constraints. The authors present the IET backbone and a Similarity-Fused FFN (SF-FFN) to fuse information among highly correlated tokens, achieving state-of-the-art SR performance under comparable computational budgets. Extensive ablations and visualizations validate the effectiveness of DLSG initialization, expansion, and sparsification, and demonstrate improved texture recovery and edge sharpness. The approach offers a scalable, efficient alternative to traditional windowed or category-based attention with potential applicability to other vision tasks requiring adaptive, asymmetric information aggregation.

Abstract

Single Image Super-Resolution (SISR) is a fundamental computer vision task that aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) input. Transformer-based methods have achieved remarkable performance by modeling long-range dependencies in degraded images. However, their feature-intensive attention computation incurs high computational cost. To improve efficiency, most existing approaches partition images into fixed groups and restrict attention within each group. Such group-wise attention overlooks the inherent asymmetry in token similarities, thereby failing to enable flexible and token-adaptive attention computation. To address this limitation, we propose the Individualized Exploratory Transformer (IET), which introduces a novel Individualized Exploratory Attention (IEA) mechanism that allows each token to adaptively select its own content-aware and independent attention candidates. This token-adaptive and asymmetric design enables more precise information aggregation while maintaining computational efficiency. Extensive experiments on standard SR benchmarks demonstrate that IET achieves state-of-the-art performance under comparable computational complexity.

Figures (12)

• Figure 1: Comparison of different attention mechanism. (a) Window-based self-attention, which groups tokens based on spatial positions and limits attention candidates within each group. (b) Category-based self-attention, which groups tokens by coarse texture while still lacks flexibility; (c) our proposed individualized exploratory attention, where tokens can adaptively and asymmetrically explore one-way neighbors.
• Figure 2: The proposed DLSG initialization strategy. In the nearby region, the attention scope includes all tokens inside, while in the distant region, one token is uniformly sampled from each $d \times d$ patch and added to the attention candidates.
• Figure 3: The proposed sparsification and expansion mechanism.
• Figure 4: The proposed individualized exploratory attention. We first apply the sparsification mechanism to prune neighbors with low similarity, and then employ the expansion mechanism to explore new two-hop neighbors.
• Figure 5: The proposed Similarity-Fused Feed-Forward Network (SF-FFN), which enables cross-channel feature interaction among semantically similar tokens.