Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequential Order-Robust Mamba for Time Series Forecasting

Seunghan Lee, Juri Hong, Kibok Lee, Taeyoung Park

TL;DR

SOR-Mamba is proposed, a TS forecasting method that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order and eliminates the 1D-convolution originally designed to capture local information in sequential data.

Abstract

Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data. However, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios. Code is available at https://github.com/seunghan96/SOR-Mamba.

Sequential Order-Robust Mamba for Time Series Forecasting

TL;DR

SOR-Mamba is proposed, a TS forecasting method that incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order and eliminates the 1D-convolution originally designed to capture local information in sequential data.

Abstract

Mamba has recently emerged as a promising alternative to Transformers, offering near-linear complexity in processing sequential data. However, while channels in time series (TS) data have no specific order in general, recent studies have adopted Mamba to capture channel dependencies (CD) in TS, introducing a sequential order bias. To address this issue, we propose SOR-Mamba, a TS forecasting method that 1) incorporates a regularization strategy to minimize the discrepancy between two embedding vectors generated from data with reversed channel orders, thereby enhancing robustness to channel order, and 2) eliminates the 1D-convolution originally designed to capture local information in sequential data. Furthermore, we introduce channel correlation modeling (CCM), a pretraining task aimed at preserving correlations between channels from the data space to the latent space in order to enhance the ability to capture CD. Extensive experiments demonstrate the efficacy of the proposed method across standard and transfer learning scenarios. Code is available at https://github.com/seunghan96/SOR-Mamba.

Paper Structure

This paper contains 26 sections, 5 equations, 29 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Methodology
  5. Architecture of SOR-Mamba
  6. Regularization with CD-Mamba Block
  7. Channel Correlation Modeling
  8. Experiments
  9. Experimental Settings
  10. Time Series Forecasting
  11. Transfer Learning
  12. Ablation Study
  13. Analysis
  14. Conclusion
  15. Dataset Statistics and Experimental Setups
  16. ...and 11 more sections

Figures (29)

  • Figure 1: Capturing CD with Mamba, which has a sequential order bias, is challenging as channels lack an inherent sequential order.
  • Figure 2: Overall framework of SOR-Mamba. (a) shows the architecture of SOR-Mamba, where the CD-Mamba block is regularized to minimize the distance between two vectors derived from reversed channel orders. (b) shows the CD-Mamba block, where the 1D-conv from the Mamba block is removed, as channels do not have a sequential order, which is further explained in Appendix \ref{['sec:1d_conv_remove']}.
  • Figure 3: Channel correlation modeling.
  • Figure 4: SL vs. SSL.
  • Figure 5: Results of transfer learning.
  • ...and 24 more figures