Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Mamba: A Comprehensive Survey and Taxonomy

Xiao Liu, Chenxu Zhang, Lei Zhang

TL;DR

This survey focuses on Mamba's application to a variety of visual tasks and data types, and discusses its predecessors, recent advances and far-reaching impact on a wide range of domains.

Abstract

State Space Model (SSM) is a mathematical model used to describe and analyze the behavior of dynamic systems. This model has witnessed numerous applications in several fields, including control theory, signal processing, economics and machine learning. In the field of deep learning, state space models are used to process sequence data, such as time series analysis, natural language processing (NLP) and video understanding. By mapping sequence data to state space, long-term dependencies in the data can be better captured. In particular, modern SSMs have shown strong representational capabilities in NLP, especially in long sequence modeling, while maintaining linear time complexity. Notably, based on the latest state-space models, Mamba merges time-varying parameters into SSMs and formulates a hardware-aware algorithm for efficient training and inference. Given its impressive efficiency and strong long-range dependency modeling capability, Mamba is expected to become a new AI architecture that may outperform Transformer. Recently, a number of works have attempted to study the potential of Mamba in various fields, such as general vision, multi-modal, medical image analysis and remote sensing image analysis, by extending Mamba from natural language domain to visual domain. To fully understand Mamba in the visual domain, we conduct a comprehensive survey and present a taxonomy study. This survey focuses on Mamba's application to a variety of visual tasks and data types, and discusses its predecessors, recent advances and far-reaching impact on a wide range of domains. Since Mamba is now on an upward trend, please actively notice us if you have new findings, and new progress on Mamba will be included in this survey in a timely manner and updated on the Mamba project at https://github.com/lx6c78/Vision-Mamba-A-Comprehensive-Survey-and-Taxonomy.

Vision Mamba: A Comprehensive Survey and Taxonomy

TL;DR

This survey focuses on Mamba's application to a variety of visual tasks and data types, and discusses its predecessors, recent advances and far-reaching impact on a wide range of domains.

Abstract

State Space Model (SSM) is a mathematical model used to describe and analyze the behavior of dynamic systems. This model has witnessed numerous applications in several fields, including control theory, signal processing, economics and machine learning. In the field of deep learning, state space models are used to process sequence data, such as time series analysis, natural language processing (NLP) and video understanding. By mapping sequence data to state space, long-term dependencies in the data can be better captured. In particular, modern SSMs have shown strong representational capabilities in NLP, especially in long sequence modeling, while maintaining linear time complexity. Notably, based on the latest state-space models, Mamba merges time-varying parameters into SSMs and formulates a hardware-aware algorithm for efficient training and inference. Given its impressive efficiency and strong long-range dependency modeling capability, Mamba is expected to become a new AI architecture that may outperform Transformer. Recently, a number of works have attempted to study the potential of Mamba in various fields, such as general vision, multi-modal, medical image analysis and remote sensing image analysis, by extending Mamba from natural language domain to visual domain. To fully understand Mamba in the visual domain, we conduct a comprehensive survey and present a taxonomy study. This survey focuses on Mamba's application to a variety of visual tasks and data types, and discusses its predecessors, recent advances and far-reaching impact on a wide range of domains. Since Mamba is now on an upward trend, please actively notice us if you have new findings, and new progress on Mamba will be included in this survey in a timely manner and updated on the Mamba project at https://github.com/lx6c78/Vision-Mamba-A-Comprehensive-Survey-and-Taxonomy.
Paper Structure (41 sections, 5 equations, 6 figures, 7 tables)

This paper contains 41 sections, 5 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Formulation of Mamba
  3. State Space Models
  4. Selective State Space Models (S6)
  5. Discussion and Summary
  6. Mamba in General Vision Tasks
  7. Mamba for High-level/Mid-level Vision
  8. Vision Backbone with Mamba
  9. Video Analysis and Understanding
  10. Vertical-domain Vision
  11. Mamba for Low-level Vision
  12. Image Denoising
  13. Image Restoration
  14. Mamba for 3D Vision
  15. Point Cloud Analysis
  16. ...and 26 more sections

Figures (6)

  • Figure 1: S4 S4 of the structural block diagram. It consists of three parts: state space models, HiPPO for handling long-range dependencies and discretization for creating recurrent and convolution representations.
  • Figure 2: Architecture of Mamba block (image from Mamba)
  • Figure 3: Comparison of different selective scan methods in Vim, VMamba, Plain Mamba, MambaAD, LocalMamba, EfficientVMamba, VideoMamba and ZigMa.
  • Figure 4: Visual state space (VSS) block (image from VMamba).
  • Figure 5: An overview of the first vision Mamba framework (Vim) (image from VisionME).
  • ...and 1 more figures