A Survey on Visual Mamba
Hanwei Zhang, Ying Zhu, Dan Wang, Lijun Zhang, Tianxiang Chen, Zi Ye
TL;DR
The paper surveys the adoption of Selective State Space Models (Mamba) for vision, addressing the quadratic complexity of self-attention in transformers by emphasizing linear-time processing and hardware-aware design. It introduces Vision Mamba variants (ViM, VSS) and 2D/3D scanning strategies, surveys how Mamba blocks serve as backbones or are combined with convolution, recurrence, and attention, and catalogs applications across general vision, medical imaging, and remote sensing. Key contributions include a taxonomy of vision Mamba models, detailed discussions of scanning mechanisms, and summaries of representative architectures and tasks, highlighting performance-efficiency trade-offs and deployment considerations. The survey underscores Mamba’s potential as a scalable alternative to transformers for long-sequence and high-resolution vision tasks, while outlining challenges such as pre-training, interpretability, robustness, and real-time deployment on resource-constrained platforms.
Abstract
State space models (SSMs) with selection mechanisms and hardware-aware architectures, namely Mamba, have recently demonstrated significant promise in long-sequence modeling. Since the self-attention mechanism in transformers has quadratic complexity with image size and increasing computational demands, the researchers are now exploring how to adapt Mamba for computer vision tasks. This paper is the first comprehensive survey aiming to provide an in-depth analysis of Mamba models in the field of computer vision. It begins by exploring the foundational concepts contributing to Mamba's success, including the state space model framework, selection mechanisms, and hardware-aware design. Next, we review these vision mamba models by categorizing them into foundational ones and enhancing them with techniques such as convolution, recurrence, and attention to improve their sophistication. We further delve into the widespread applications of Mamba in vision tasks, which include their use as a backbone in various levels of vision processing. This encompasses general visual tasks, Medical visual tasks (e.g., 2D / 3D segmentation, classification, and image registration, etc.), and Remote Sensing visual tasks. We specially introduce general visual tasks from two levels: High/Mid-level vision (e.g., Object detection, Segmentation, Video classification, etc.) and Low-level vision (e.g., Image super-resolution, Image restoration, Visual generation, etc.). We hope this endeavor will spark additional interest within the community to address current challenges and further apply Mamba models in computer vision.