Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series

Badri N. Patro, Vijay S. Agneeswaran

TL;DR

SiMBA introduces EinFFT as a stable, frequency-domain channel mixer and combines it with a Mamba-based sequence model to form a scalable, inductive-bias-friendly architecture for long sequences. By stabilizing the Mamba dynamics and leveraging spectral processing, SiMBA closes much of the performance gap between state-space models and attention-based transformers on ImageNet and multivariate time-series benchmarks. Across vision and time-series experiments, SiMBA achieves competitive ImageNet top-1 accuracy and state-of-the-art results among SSMs on several datasets, while offering strong transfer and task-learning capabilities. The work highlights the practical impact of frequency-domain channel mixing for cross-domain sequence modeling and paves the way for further architectural variants within the SiMBA framework.

Abstract

Transformers have widely adopted attention networks for sequence mixing and MLPs for channel mixing, playing a pivotal role in achieving breakthroughs across domains. However, recent literature highlights issues with attention networks, including low inductive bias and quadratic complexity concerning input sequence length. State Space Models (SSMs) like S4 and others (Hippo, Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address the above issues to help handle longer sequence lengths. Mamba, while being the state-of-the-art SSM, has a stability issue when scaled to large networks for computer vision datasets. We propose SiMBA, a new architecture that introduces Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations and uses the Mamba block for sequence modeling. Extensive performance studies across image and time-series benchmarks demonstrate that SiMBA outperforms existing SSMs, bridging the performance gap with state-of-the-art transformers. Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet and transfer learning benchmarks such as Stanford Car and Flower as well as task learning benchmarks as well as seven time series benchmark datasets. The project page is available on this website ~\url{https://github.com/badripatro/Simba}.

SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series

TL;DR

SiMBA introduces EinFFT as a stable, frequency-domain channel mixer and combines it with a Mamba-based sequence model to form a scalable, inductive-bias-friendly architecture for long sequences. By stabilizing the Mamba dynamics and leveraging spectral processing, SiMBA closes much of the performance gap between state-space models and attention-based transformers on ImageNet and multivariate time-series benchmarks. Across vision and time-series experiments, SiMBA achieves competitive ImageNet top-1 accuracy and state-of-the-art results among SSMs on several datasets, while offering strong transfer and task-learning capabilities. The work highlights the practical impact of frequency-domain channel mixing for cross-domain sequence modeling and paves the way for further architectural variants within the SiMBA framework.

Abstract

Transformers have widely adopted attention networks for sequence mixing and MLPs for channel mixing, playing a pivotal role in achieving breakthroughs across domains. However, recent literature highlights issues with attention networks, including low inductive bias and quadratic complexity concerning input sequence length. State Space Models (SSMs) like S4 and others (Hippo, Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address the above issues to help handle longer sequence lengths. Mamba, while being the state-of-the-art SSM, has a stability issue when scaled to large networks for computer vision datasets. We propose SiMBA, a new architecture that introduces Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations and uses the Mamba block for sequence modeling. Extensive performance studies across image and time-series benchmarks demonstrate that SiMBA outperforms existing SSMs, bridging the performance gap with state-of-the-art transformers. Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet and transfer learning benchmarks such as Stanford Car and Flower as well as task learning benchmarks as well as seven time series benchmark datasets. The project page is available on this website ~\url{https://github.com/badripatro/Simba}.
Paper Structure (25 sections, 16 equations, 3 figures, 10 tables, 1 algorithm)

This paper contains 25 sections, 16 equations, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Channel Mixing: Einstein FFT (EinFFT)
  5. Spectral Transformation
  6. Frequency-domain Channel Mixing
  7. Inverse Spectral Transformation
  8. EinFFT Architecture for SiMBA
  9. Sequence Modeling: Mamba based SSM
  10. Discrete-time SSM:
  11. Convolutional Kernel Representation
  12. Experiment
  13. SOTA Comparison on ImageNet 1K
  14. SOTA for Multi-Variate Time Series Forecasting
  15. Ablation Analysis of Model SiMBA
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Simplified Mamba Based Architecture.
  • Figure 2: Comparison of ImageNet Top-1 Accuracy (%) vs Params(M) of Transformer architectures and State space architectures. SiMBA is a better state space model compared to other state space models for Vision data.
  • Figure 3: Comparison of ImageNet Top-1 Accuracy (%) vs Params(M) of State space architectures. SiMBA is a better state space model compared to other state space models for Vision data.