Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution

Zhenyu Hu, Wanjie Sun

TL;DR

Unifying Dimensions develops LAMNet, a convolution-based Transformer for lightweight SISR that replaces local self-attention with a Linear Adaptive Mixer built from LSAM and CSM. A parameter-free Information Exchange Module and a Dual-Gated FFN preserve cross-branch interactions and high-dimensional channel information, yielding linear-complexity adaptive modeling. Empirically, LAMNet achieves Transformer-like performance with 2–3× faster inference than SA-based transformers and outperforms prior lightweight SR methods on multiple benchmarks. The work offers practical implications for real-time SR on devices with restricted compute and memory resources by balancing modeling power with throughput.

Abstract

Window-based transformers have demonstrated outstanding performance in super-resolution tasks due to their adaptive modeling capabilities through local self-attention (SA). However, they exhibit higher computational complexity and inference latency than convolutional neural networks. In this paper, we first identify that the adaptability of the Transformers is derived from their adaptive spatial aggregation and advanced structural design, while their high latency results from the computational costs and memory layout transformations associated with the local SA. To simulate this aggregation approach, we propose an effective convolution-based linear focal separable attention (FSA), allowing for long-range dynamic modeling with linear complexity. Additionally, we introduce an effective dual-branch structure combined with an ultra-lightweight information exchange module (IEM) to enhance the aggregation of information by the Token Mixer. Finally, with respect to the structure, we modify the existing spatial-gate-based feedforward neural networks by incorporating a self-gate mechanism to preserve high-dimensional channel information, enabling the modeling of more complex relationships. With these advancements, we construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet). Extensive experiments demonstrate that LAMNet achieves better performance than existing SA-based Transformer methods while maintaining the computational efficiency of convolutional neural networks, which can achieve a \(3\times\) speedup of inference time. The code will be publicly available at: https://github.com/zononhzy/LAMNet.

Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution

TL;DR

Unifying Dimensions develops LAMNet, a convolution-based Transformer for lightweight SISR that replaces local self-attention with a Linear Adaptive Mixer built from LSAM and CSM. A parameter-free Information Exchange Module and a Dual-Gated FFN preserve cross-branch interactions and high-dimensional channel information, yielding linear-complexity adaptive modeling. Empirically, LAMNet achieves Transformer-like performance with 2–3× faster inference than SA-based transformers and outperforms prior lightweight SR methods on multiple benchmarks. The work offers practical implications for real-time SR on devices with restricted compute and memory resources by balancing modeling power with throughput.

Abstract

Window-based transformers have demonstrated outstanding performance in super-resolution tasks due to their adaptive modeling capabilities through local self-attention (SA). However, they exhibit higher computational complexity and inference latency than convolutional neural networks. In this paper, we first identify that the adaptability of the Transformers is derived from their adaptive spatial aggregation and advanced structural design, while their high latency results from the computational costs and memory layout transformations associated with the local SA. To simulate this aggregation approach, we propose an effective convolution-based linear focal separable attention (FSA), allowing for long-range dynamic modeling with linear complexity. Additionally, we introduce an effective dual-branch structure combined with an ultra-lightweight information exchange module (IEM) to enhance the aggregation of information by the Token Mixer. Finally, with respect to the structure, we modify the existing spatial-gate-based feedforward neural networks by incorporating a self-gate mechanism to preserve high-dimensional channel information, enabling the modeling of more complex relationships. With these advancements, we construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet). Extensive experiments demonstrate that LAMNet achieves better performance than existing SA-based Transformer methods while maintaining the computational efficiency of convolutional neural networks, which can achieve a speedup of inference time. The code will be publicly available at: https://github.com/zononhzy/LAMNet.
Paper Structure (15 sections, 11 equations, 10 figures, 4 tables)

This paper contains 15 sections, 11 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Lightweight SISR Model
  4. Feed-Forward Network
  5. Methodology
  6. Linear Adaptive Mixer Network
  7. Unified Linear Mixer
  8. Dual-Group Feed-Forward Network
  9. Complexity Analysis of ULM and DGFN
  10. Experiments
  11. Experimental Settings
  12. Benchmarking
  13. Ablation Study
  14. Limitations
  15. Conclusion

Figures (10)

  • Figure 1: Comparison of different operators. Separable convolutions utilize one-dimensional kernels to achieve linear complexity in feature processing, which is inflexible. The local self-attention mechanism adaptively generates weights for query tokens, maintaining high computational complexity. The separable attention, while retaining its adaptive nature, linearly generates sparse weights to handle super-resolution tasks effectively.
  • Figure 2: The overall architecture of the proposed Linear Adaptive Mixer Network (LAMNet) is presented, where the core operation, highlighted in the blue area, is the Linear Adaptive Mixer (LAM) Block. The Unified Linear Mixer (ULM), Dual-Gated Feed-Forward Network (DGFN), and LayerNorm constitute the basic Transformer block, which is stacked $n$ times to form the main structure of the LAM block. Features are processed within the model to optimize efficiency using the $C\times H\times W$ memory layout.
  • Figure 3: The framework of Unified Linear Mixer (ULM), which incorporates Linear-Spatial Adaptive mixer (LSAM) for spatial components and Channel Selective Mixer (CSM) for channel components. Spatial information is then continuously aggregated in horizontal and vertical directions using the Focal Seperable Attention (FSA). Subsequently, the two Information Exchange Modules facilitate information interaction between the spatial and channel branches.
  • Figure 4: The overall structure of the Dual-Gated Feed-Forward Network includes a self-gate and cross-gate operation to enhance the channel dimension and mitigate the gate's impact on it.
  • Figure 5: Visual comparisons on Urban100 and Manga109 with scale factor 4.
  • ...and 5 more figures