Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting

Xiuding Cai, Yaoyao Zhu, Xueyao Wang, Yu Yao

TL;DR

This paper tackles long-term time series forecasting by addressing Transformer limitations such as quadratic complexity and permutation bias. It introduces MambaTS, an enhanced selective State Space Model framework that adds Variable Scan along Time (VST), a Temporal Mamba Block (TMB) without causal convolution, dropout on selective parameters, and a Variable-Aware Scan along Time (VAST) with Variable Permutation Training (VPT). Through these innovations, MambaTS achieves state-of-the-art results on eight public multivariate datasets while maintaining linear computational complexity with respect to sequence length, approximately $\,\mathcal{O}(\frac{KL}{P})$. The approach demonstrates improved modeling of global temporal and inter-variable dependencies, with VAST-guided inference and SA-based ATSP solving enabling dynamic, data-driven scan orders. Overall, MambaTS offers a scalable, accurate alternative for LTSF in settings with many variables and long horizons.

Abstract

In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), but faces challenges such as quadratic complexity and permutation invariant bias. A recent model, Mamba, based on selective state space models (SSMs), has emerged as a competitive alternative to Transformer, offering comparable performance with higher throughput and linear complexity related to sequence length. In this study, we analyze the limitations of current Mamba in LTSF and propose four targeted improvements, leading to MambaTS. We first introduce variable scan along time to arrange the historical information of all the variables together. We suggest that causal convolution in Mamba is not necessary for LTSF and propose the Temporal Mamba Block (TMB). We further incorporate a dropout mechanism for selective parameters of TMB to mitigate model overfitting. Moreover, we tackle the issue of variable scan order sensitivity by introducing variable permutation training. We further propose variable-aware scan along time to dynamically discover variable relationships during training and decode the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance.

MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting

TL;DR

This paper tackles long-term time series forecasting by addressing Transformer limitations such as quadratic complexity and permutation bias. It introduces MambaTS, an enhanced selective State Space Model framework that adds Variable Scan along Time (VST), a Temporal Mamba Block (TMB) without causal convolution, dropout on selective parameters, and a Variable-Aware Scan along Time (VAST) with Variable Permutation Training (VPT). Through these innovations, MambaTS achieves state-of-the-art results on eight public multivariate datasets while maintaining linear computational complexity with respect to sequence length, approximately . The approach demonstrates improved modeling of global temporal and inter-variable dependencies, with VAST-guided inference and SA-based ATSP solving enabling dynamic, data-driven scan orders. Overall, MambaTS offers a scalable, accurate alternative for LTSF in settings with many variables and long horizons.

Abstract

In recent years, Transformers have become the de-facto architecture for long-term sequence forecasting (LTSF), but faces challenges such as quadratic complexity and permutation invariant bias. A recent model, Mamba, based on selective state space models (SSMs), has emerged as a competitive alternative to Transformer, offering comparable performance with higher throughput and linear complexity related to sequence length. In this study, we analyze the limitations of current Mamba in LTSF and propose four targeted improvements, leading to MambaTS. We first introduce variable scan along time to arrange the historical information of all the variables together. We suggest that causal convolution in Mamba is not necessary for LTSF and propose the Temporal Mamba Block (TMB). We further incorporate a dropout mechanism for selective parameters of TMB to mitigate model overfitting. Moreover, we tackle the issue of variable scan order sensitivity by introducing variable permutation training. We further propose variable-aware scan along time to dynamically discover variable relationships during training and decode the optimal variable scan order by solving the shortest path visiting all nodes problem during inference. Extensive experiments conducted on eight public datasets demonstrate that MambaTS achieves new state-of-the-art performance.
Paper Structure (33 sections, 7 equations, 6 figures, 6 tables)

This paper contains 33 sections, 7 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long-term time series forecasting
  4. State Space Models
  5. Preliminaries
  6. State Space Models
  7. Discretization
  8. Selective Scan Mechanism
  9. Model architecture
  10. Overall Architecture
  11. Patching and Tokenization
  12. Variable Scan along Time
  13. Encoder
  14. Prediction Head
  15. Instance Normalization
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Case study: Mamba for LTSF tasks.
  • Figure 2: The overall architecture of MambaTS.
  • Figure 3: Dropout ablations of TMB. Left: TMB with varying dropout rates on Weather dataset. Right: Loss curves over training.
  • Figure 4: Sampling results of VAST. Employing swarm plot to prevent overlapping points (better in color).
  • Figure 5: Ablations on VAST.
  • ...and 1 more figures