Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling

Mahdi Karami, Ali Behrouz, Peilin Zhong, Razvan Pascanu, Vahab Mirrokni

TL;DR

MS-SSM introduces a multi-resolution state-space model that decomposes sequences into multiple scales using a SWT-based decomposition, with an array of parallel SSMs processing each scale. Outputs are fused through an input-dependent scale-mixer, aided by scale-aware initialization and gating to balance memory across resolutions, while maintaining computational efficiency via convolutional and FFT-based techniques. Empirically, MS-SSM delivers state-of-the-art or competitive performance across image classification (ImageNet-1K, sCIFAR), time-series ECG (PTB-XL), and hierarchical reasoning (ListOps), and shows a meaningful long-range boost on the Long Range Arena benchmark (approximately a 14.4% gain over Mamba). The work demonstrates that multi-scale processing within state-space architectures yields improved long-range modeling with scalable, parallelizable computation, suggesting broad applicability including potential NLP extensions and memory analyses for RNN/SSM models.

Abstract

State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, enabling fast inference, parallelizable training, and control over recurrence stability. However, traditional SSMs often suffer from limited effective memory, requiring larger state sizes for improved recall. Moreover, existing SSMs struggle to capture multi-scale dependencies, which are essential for modeling complex structures in time series, images, and natural language. This paper introduces a multi-scale SSM framework that addresses these limitations by representing sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics. By capturing both fine-grained, high-frequency patterns and coarse, global trends, MS-SSM enhances memory efficiency and long-range modeling. We further introduce an input-dependent scale-mixer, enabling dynamic information fusion across resolutions. The proposed approach significantly improves sequence modeling, particularly in long-range and hierarchical tasks, while maintaining computational efficiency. Extensive experiments on benchmarks, including Long Range Arena, hierarchical reasoning, time series classification, and image recognition, demonstrate that MS-SSM consistently outperforms prior SSM-based models, highlighting the benefits of multi-resolution processing in state-space architectures.

MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling

TL;DR

MS-SSM introduces a multi-resolution state-space model that decomposes sequences into multiple scales using a SWT-based decomposition, with an array of parallel SSMs processing each scale. Outputs are fused through an input-dependent scale-mixer, aided by scale-aware initialization and gating to balance memory across resolutions, while maintaining computational efficiency via convolutional and FFT-based techniques. Empirically, MS-SSM delivers state-of-the-art or competitive performance across image classification (ImageNet-1K, sCIFAR), time-series ECG (PTB-XL), and hierarchical reasoning (ListOps), and shows a meaningful long-range boost on the Long Range Arena benchmark (approximately a 14.4% gain over Mamba). The work demonstrates that multi-scale processing within state-space architectures yields improved long-range modeling with scalable, parallelizable computation, suggesting broad applicability including potential NLP extensions and memory analyses for RNN/SSM models.

Abstract

State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling. They rely on linear recurrences to integrate information over time, enabling fast inference, parallelizable training, and control over recurrence stability. However, traditional SSMs often suffer from limited effective memory, requiring larger state sizes for improved recall. Moreover, existing SSMs struggle to capture multi-scale dependencies, which are essential for modeling complex structures in time series, images, and natural language. This paper introduces a multi-scale SSM framework that addresses these limitations by representing sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics. By capturing both fine-grained, high-frequency patterns and coarse, global trends, MS-SSM enhances memory efficiency and long-range modeling. We further introduce an input-dependent scale-mixer, enabling dynamic information fusion across resolutions. The proposed approach significantly improves sequence modeling, particularly in long-range and hierarchical tasks, while maintaining computational efficiency. Extensive experiments on benchmarks, including Long Range Arena, hierarchical reasoning, time series classification, and image recognition, demonstrate that MS-SSM consistently outperforms prior SSM-based models, highlighting the benefits of multi-resolution processing in state-space architectures.
Paper Structure (23 sections, 9 equations, 5 figures, 7 tables)

This paper contains 23 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. State Space Models
  4. SSM.
  5. Multi-Scale Decomposition
  6. Experiments
  7. Effective Receptive Field
  8. Conclusion and Discussion
  9. Details
  10. Notation definition
  11. Model Architecture
  12. Scale Mixing
  13. Effective Receptive Field
  14. Efficient Implementation of Multi-scale Decomposition Layer.
  15. Experimental Details
  16. ...and 8 more sections

Figures (5)

  • Figure 1: (a) Block diagram of the MS-SSM model. (b) Multi-scale convolution layer. The multi-scale conv layer, which decomposes the signal into multiple scales, is composed of nested convolution layers $\texttt{Conv1d}$ defined in (\ref{['eq:conv1Dscale']}). The scale-mixer combines the scales through an input-dependent weighted summation defined in (\ref{['eq:scaleMixer']}).
  • Figure 2: Initialization scheme for 3 different scales with $N=3$ and $\Delta_0=0.2$ .
  • Figure : (a)
  • Figure : (a)
  • Figure : (b)