Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MxT: Mamba x Transformer for Image Inpainting

Shuang Chen, Amir Atapour-Abarghouei, Haozheng Zhang, Hubert P. H. Shum

TL;DR

This work tackles image inpainting by addressing the need for both local texture fidelity and global context. It introduces MxT, a hybrid architecture that fuses Mamba's linear-cost, pixel-level long-range modeling with transformer-based global interactions through a Hybrid Module, enabling dual-level (pixel and patch) information exchange. Empirical results on CelebA-HQ and Places2 show MxT achieving state-of-the-art or competitive performance across mask ratios, while maintaining linear complexity that scales to high-resolution images. The approach demonstrates practical impact for efficient, high-quality inpainting and is complemented by a plan to release code and explore multimodal guidance, e.g., CLIP-based control.

Abstract

Image inpainting, or image completion, is a crucial task in computer vision that aims to restore missing or damaged regions of images with semantically coherent content. This technique requires a precise balance of local texture replication and global contextual understanding to ensure the restored image integrates seamlessly with its surroundings. Traditional methods using Convolutional Neural Networks (CNNs) are effective at capturing local patterns but often struggle with broader contextual relationships due to the limited receptive fields. Recent advancements have incorporated transformers, leveraging their ability to understand global interactions. However, these methods face computational inefficiencies and struggle to maintain fine-grained details. To overcome these challenges, we introduce MxT composed of the proposed Hybrid Module (HM), which combines Mamba with the transformer in a synergistic manner. Mamba is adept at efficiently processing long sequences with linear computational costs, making it an ideal complement to the transformer for handling long-scale data interactions. Our HM facilitates dual-level interaction learning at both pixel and patch levels, greatly enhancing the model to reconstruct images with high quality and contextual accuracy. We evaluate MxT on the widely-used CelebA-HQ and Places2-standard datasets, where it consistently outperformed existing state-of-the-art methods. The code will be released: {\url{https://github.com/ChrisChen1023/MxT}}.

MxT: Mamba x Transformer for Image Inpainting

TL;DR

This work tackles image inpainting by addressing the need for both local texture fidelity and global context. It introduces MxT, a hybrid architecture that fuses Mamba's linear-cost, pixel-level long-range modeling with transformer-based global interactions through a Hybrid Module, enabling dual-level (pixel and patch) information exchange. Empirical results on CelebA-HQ and Places2 show MxT achieving state-of-the-art or competitive performance across mask ratios, while maintaining linear complexity that scales to high-resolution images. The approach demonstrates practical impact for efficient, high-quality inpainting and is complemented by a plan to release code and explore multimodal guidance, e.g., CLIP-based control.

Abstract

Image inpainting, or image completion, is a crucial task in computer vision that aims to restore missing or damaged regions of images with semantically coherent content. This technique requires a precise balance of local texture replication and global contextual understanding to ensure the restored image integrates seamlessly with its surroundings. Traditional methods using Convolutional Neural Networks (CNNs) are effective at capturing local patterns but often struggle with broader contextual relationships due to the limited receptive fields. Recent advancements have incorporated transformers, leveraging their ability to understand global interactions. However, these methods face computational inefficiencies and struggle to maintain fine-grained details. To overcome these challenges, we introduce MxT composed of the proposed Hybrid Module (HM), which combines Mamba with the transformer in a synergistic manner. Mamba is adept at efficiently processing long sequences with linear computational costs, making it an ideal complement to the transformer for handling long-scale data interactions. Our HM facilitates dual-level interaction learning at both pixel and patch levels, greatly enhancing the model to reconstruct images with high quality and contextual accuracy. We evaluate MxT on the widely-used CelebA-HQ and Places2-standard datasets, where it consistently outperformed existing state-of-the-art methods. The code will be released: {\url{https://github.com/ChrisChen1023/MxT}}.
Paper Structure (16 sections, 10 equations, 4 figures, 3 tables)

This paper contains 16 sections, 10 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Inpainting
  4. SSMs in Computer Vision
  5. Preliminary
  6. State Space Models and Mamba
  7. Method
  8. Hybrid Module
  9. Loss Functions
  10. Experiment Results
  11. Comparison with State of the Art
  12. Ablation Study
  13. Application: High-Resolution Image Inpainting
  14. Limitation and Discussion
  15. Conclusion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: (a) The architecture overview of the proposed M$\times$T. (b) The Hybrid Block is composed of $n$ proposed Hybrid Modules. (c) The proposed Hybrid Module, consisted of a Mamba Block, a Spatial Reduced Self-Attention and a Context Broadcasting Feed-forward Network. (d) The Spatial Reduced Self-Attention provides spatial awareness. (e) The Mamba Block captures pixel-level interaction. (f) The Context Broadcasting Feed-forward Network transfers the features.
  • Figure 2: Visual comparisons at $(256 \times 256)$ resolution against the state-of-the-art methods on CelebA-HQ karras2017progressive (first two rows) and Places2 zhou2017places (last two rows).
  • Figure 3: Illustration of the application on real-world high-resolution images with resolution of $2560 \times 1920$.
  • Figure 4: Illustration of a fail case with a much larger mask ratio.