Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mamba-Adaptor: State Space Model Adaptor for Visual Recognition

Fei Xie, Jiahao Nie, Yujin Tang, Wenkang Zhang, Hongshen Zhao

TL;DR

This work introduces Mamba-Adaptor, a plug-and-play enhancement for Vision Mamba models that addresses temporal forgetting and weak spatial priors by coupling Adaptor-T (learnable memory selection for long-range retention) with Adaptor-S (multi-scale depthwise spatial aggregation). The two modules are designed to be inserted in a hardware-friendly way, offering parallel or sequential integration for transfer learning or from-scratch training. Empirical results across ImageNet-1K and COCO show state-of-the-art or competitive performance, with booster and transfer-learning experiments demonstrating efficient improvements using modest parameter追加. Collectively, Mamba-Adaptor provides a practical pathway to leverage SSMs for vision tasks with improved context modeling and spatial priors, while maintaining computational efficiency.

Abstract

Recent State Space Models (SSM), especially Mamba, have demonstrated impressive performance in visual modeling and possess superior model efficiency. However, the application of Mamba to visual tasks suffers inferior performance due to three main constraints existing in the sequential model: 1) Casual computing is incapable of accessing global context; 2) Long-range forgetting when computing the current hidden states; 3) Weak spatial structural modeling due to the transformed sequential input. To address these issues, we investigate a simple yet powerful vision task Adaptor for Mamba models, which consists of two functional modules: Adaptor-T and Adaptor-S. When solving the hidden states for SSM, we apply a lightweight prediction module Adaptor-T to select a set of learnable locations as memory augmentations to ease long-range forgetting issues. Moreover, we leverage Adapator-S, composed of multi-scale dilated convolutional kernels, to enhance the spatial modeling and introduce the image inductive bias into the feature output. Both modules can enlarge the context modeling in casual computing, as the output is enhanced by the inaccessible features. We explore three usages of Mamba-Adaptor: A general visual backbone for various vision tasks; A booster module to raise the performance of pretrained backbones; A highly efficient fine-tuning module that adapts the base model for transfer learning tasks. Extensive experiments verify the effectiveness of Mamba-Adaptor in three settings. Notably, our Mamba-Adaptor achieves state-of the-art performance on the ImageNet and COCO benchmarks.

Mamba-Adaptor: State Space Model Adaptor for Visual Recognition

TL;DR

This work introduces Mamba-Adaptor, a plug-and-play enhancement for Vision Mamba models that addresses temporal forgetting and weak spatial priors by coupling Adaptor-T (learnable memory selection for long-range retention) with Adaptor-S (multi-scale depthwise spatial aggregation). The two modules are designed to be inserted in a hardware-friendly way, offering parallel or sequential integration for transfer learning or from-scratch training. Empirical results across ImageNet-1K and COCO show state-of-the-art or competitive performance, with booster and transfer-learning experiments demonstrating efficient improvements using modest parameter追加. Collectively, Mamba-Adaptor provides a practical pathway to leverage SSMs for vision tasks with improved context modeling and spatial priors, while maintaining computational efficiency.

Abstract

Recent State Space Models (SSM), especially Mamba, have demonstrated impressive performance in visual modeling and possess superior model efficiency. However, the application of Mamba to visual tasks suffers inferior performance due to three main constraints existing in the sequential model: 1) Casual computing is incapable of accessing global context; 2) Long-range forgetting when computing the current hidden states; 3) Weak spatial structural modeling due to the transformed sequential input. To address these issues, we investigate a simple yet powerful vision task Adaptor for Mamba models, which consists of two functional modules: Adaptor-T and Adaptor-S. When solving the hidden states for SSM, we apply a lightweight prediction module Adaptor-T to select a set of learnable locations as memory augmentations to ease long-range forgetting issues. Moreover, we leverage Adapator-S, composed of multi-scale dilated convolutional kernels, to enhance the spatial modeling and introduce the image inductive bias into the feature output. Both modules can enlarge the context modeling in casual computing, as the output is enhanced by the inaccessible features. We explore three usages of Mamba-Adaptor: A general visual backbone for various vision tasks; A booster module to raise the performance of pretrained backbones; A highly efficient fine-tuning module that adapts the base model for transfer learning tasks. Extensive experiments verify the effectiveness of Mamba-Adaptor in three settings. Notably, our Mamba-Adaptor achieves state-of the-art performance on the ImageNet and COCO benchmarks.
Paper Structure (38 sections, 10 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 38 sections, 10 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. State Space Model.
  4. Visual adaptor.
  5. Method
  6. Preliminaries
  7. State Space Model.
  8. Discretization of SSM.
  9. Selective Scan Mechanism.
  10. Sequence Transform for Vision Mamba.
  11. Formulation of Mamba-Adaptor
  12. Hidden State.
  13. Final Output.
  14. Adaptor-T
  15. Learnable Memory Selection.
  16. ...and 23 more sections

Figures (4)

  • Figure 1: Our Mamba-adaptor has multiple usages. It can serve as a general visual backbone network, a booster module for pre-trained backbone, and an adaptor for transfer learning.
  • Figure 2: Mamba-Adaptor consists of two modules for solving SSM mambas5: Adaptor-T conducts memory retention for the hidden states; Adaptor-S conducts spatial aggregation for final output.
  • Figure 3: Details of the Mamba-Adapter. (a) illustrates a typical architecture of vision Mamba layer vmamba, including SS2D block, FFN attention and two residual connections; (b) details of SS2D layer, which transforms the 2D image into sequences and process by SSM equation; (c) recursive solving procedure of SSM with two main steps; (d) Mamba-Adaptor with Adaptor-T and Adaptor-S in solving SSM process. FFN and LN denote feed-forward network attn and layer normalization LN.
  • Figure 4: Illustration of the parallel and sequential insertion form of Mamba-Adaptor.