MatIR: A Hybrid Mamba-Transformer Image Restoration Model

TL;DR

MatIR addresses image restoration by uniting Mamba's linear-time state-space processing with Transformer-based contextual learning. The architecture interleaves Transformer and Mamba layers, augmented by IRSS for long-range sequencing and attention-enhancement modules TWLA and CGA, followed by a reconstruction stage. Across SR, denoising, and deblurring tasks, MatIR delivers state-of-the-art results with improved efficiency and reduced memory footprint, validating the practical value of hybrid Mamba-Transformer designs. This work highlights a promising direction for high-resolution restoration where global context and scalable computation must coexist.

Abstract

In recent years, Transformers-based models have made significant progress in the field of image restoration by leveraging their inherent ability to capture complex contextual features. Recently, Mamba models have made a splash in the field of computer vision due to their ability to handle long-range dependencies and their significant computational efficiency compared to Transformers. However, Mamba currently lags behind Transformers in contextual learning capabilities. To overcome the limitations of these two models, we propose a Mamba-Transformer hybrid image restoration model called MatIR. Specifically, MatIR cross-cycles the blocks of the Transformer layer and the Mamba layer to extract features, thereby taking full advantage of the advantages of the two architectures. In the Mamba module, we introduce the Image Inpainting State Space (IRSS) module, which traverses along four scan paths to achieve efficient processing of long sequence data. In the Transformer module, we combine triangular window-based local attention with channel-based global attention to effectively activate the attention mechanism over a wider range of image pixels. Extensive experimental results and ablation studies demonstrate the effectiveness of our approach.

Figures (8)

• Figure 1: (a) Visualization of the effective receptive field (ERF) of the Transformer basic model luo2016understandingding2022scaling. The figure below shows the advantage of being larger than Manba in the context neighbor receptive field. (b) Visualization of the ERF of the Manba basic model liu2024vmamba. The figure below shows the advantage of being larger than Transformer in long sequence linear receptive fields. (c) Our proposed MatIR: A Hybrid Mamba-Transformer Image Restoration Model achieves a more significant effective receptive field.
• Figure 1: Ablation experiments for different design choices of MatIR.
• Figure 2: Overall network architecture of our MatIR.
• Figure 3: More details on IRSS, the core component of our MatIR model.
• Figure 4: Comparison of iterative performance (PSNR in dB) of the proposed MatIR Top-Left: Performance comparison of triangle local attention, channel global attention and image recovery space modules. Top-Right: Triangle local attention and channel global attention, various interval sizes, Bottom left: Different channel lengths. [On BSD100($\times4$), epoch 70]and bottom right:Performance evaluation of small, medium and large MatIR models.