Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scalable Visual State Space Model with Fractal Scanning

Lv Tang, HaoKe Xiao, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Bo Li

TL;DR

To address Transformer inefficiency on long sequences and high-res images, the paper introduces fractal scanning-based patch serialization for State Space Models (SSMs) in vision. The FractalMamba backbone uses Hilbert fractal curves with a shifting operation and a Selective SSM to capture spatial relationships, enabling linear-time inference and better multi-scale modeling. Empirical results across ImageNet, COCO, and ADE20K demonstrate competitive or superior performance versus ViTs and other SSM-based backbones, especially at high resolutions. This work advances scalable CV modeling with fractal-aware serialization, suggesting broader exploration of fractal scanning methods for SSMs.

Abstract

Foundational models have significantly advanced in natural language processing (NLP) and computer vision (CV), with the Transformer architecture becoming a standard backbone. However, the Transformer's quadratic complexity poses challenges for handling longer sequences and higher resolution images. To address this challenge, State Space Models (SSMs) like Mamba have emerged as efficient alternatives, initially matching Transformer performance in NLP tasks and later surpassing Vision Transformers (ViTs) in various CV tasks. To improve the performance of SSMs, one crucial aspect is effective serialization of image patches. Existing methods, relying on linear scanning curves, often fail to capture complex spatial relationships and produce repetitive patterns, leading to biases. To address these limitations, we propose using fractal scanning curves for patch serialization. Fractal curves maintain high spatial proximity and adapt to different image resolutions, avoiding redundancy and enhancing SSMs' ability to model complex patterns accurately. We validate our method in image classification, detection, and segmentation tasks, and the superior performance validates its effectiveness.

Scalable Visual State Space Model with Fractal Scanning

TL;DR

To address Transformer inefficiency on long sequences and high-res images, the paper introduces fractal scanning-based patch serialization for State Space Models (SSMs) in vision. The FractalMamba backbone uses Hilbert fractal curves with a shifting operation and a Selective SSM to capture spatial relationships, enabling linear-time inference and better multi-scale modeling. Empirical results across ImageNet, COCO, and ADE20K demonstrate competitive or superior performance versus ViTs and other SSM-based backbones, especially at high resolutions. This work advances scalable CV modeling with fractal-aware serialization, suggesting broader exploration of fractal scanning methods for SSMs.

Abstract

Foundational models have significantly advanced in natural language processing (NLP) and computer vision (CV), with the Transformer architecture becoming a standard backbone. However, the Transformer's quadratic complexity poses challenges for handling longer sequences and higher resolution images. To address this challenge, State Space Models (SSMs) like Mamba have emerged as efficient alternatives, initially matching Transformer performance in NLP tasks and later surpassing Vision Transformers (ViTs) in various CV tasks. To improve the performance of SSMs, one crucial aspect is effective serialization of image patches. Existing methods, relying on linear scanning curves, often fail to capture complex spatial relationships and produce repetitive patterns, leading to biases. To address these limitations, we propose using fractal scanning curves for patch serialization. Fractal curves maintain high spatial proximity and adapt to different image resolutions, avoiding redundancy and enhancing SSMs' ability to model complex patterns accurately. We validate our method in image classification, detection, and segmentation tasks, and the superior performance validates its effectiveness.
Paper Structure (14 sections, 4 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 14 sections, 4 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision Backbone Architecture
  4. State Space Model
  5. Method
  6. Preliminaries
  7. Limitations of Existing Scanning Mechanisms
  8. Fractal Scanning Mechanisms
  9. Experiments
  10. Image Classification
  11. Object Detection
  12. Semantic Segmentation
  13. Ablation Studies
  14. Conclusion

Figures (3)

  • Figure 1: The architecture of our proposed FractalMamba Backbone. The FractalMamba block contains four curves in 4 directions. Taking $8 \times 8$ image patches for example, the first curve is from (0,0) to (7,0), the second curve is from (0,0) to (0,7), the third curve is from (7,7) to (7,0), and the fourth curve is from (7,7) to (0,7).
  • Figure 2: The Hilbert curve and the corresponding shifting operation.
  • Figure 3: The variations in classification accuracy and computational complexity (FLOPs) as the resolutions of test images increase progressively.