Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention Mamba: Time Series Modeling with Adaptive Pooling Acceleration and Receptive Field Enhancements

Sijie Xiong, Shuqing Liu, Cheng Tang, Fumiya Okubo, Haoling Xiong, Atsushi Shimada

TL;DR

Attention Mamba tackles the limitations of attention and receptive field constraints in time series forecasting by integrating an Adaptive Pooling block that accelerates attention while injecting global information and a bidirectional Mamba block to enhance long-short dependencies. The framework yields improved nonlinear dependency extraction and expanded receptive fields, leading to state-of-the-art performance across diverse real-world datasets. Extensive experiments and ranking analyses demonstrate robust advantages over Transformer-based, Linear-based, and other Mamba variants, with a favorable accuracy-to-complexity trade-off. The work offers practical guidance for efficient, nonlinear, long-range dependency modeling in time series and provides code on GitHub.

Abstract

"This work has been submitted to the lEEE for possible publication. Copyright may be transferred without noticeafter which this version may no longer be accessible." Time series modeling serves as the cornerstone of real-world applications, such as weather forecasting and transportation management. Recently, Mamba has become a promising model that combines near-linear computational complexity with high prediction accuracy in time series modeling, while facing challenges such as insufficient modeling of nonlinear dependencies in attention and restricted receptive fields caused by convolutions. To overcome these limitations, this paper introduces an innovative framework, Attention Mamba, featuring a novel Adaptive Pooling block that accelerates attention computation and incorporates global information, effectively overcoming the constraints of limited receptive fields. Furthermore, Attention Mamba integrates a bidirectional Mamba block, efficiently capturing long-short features and transforming inputs into the Value representations for attention mechanisms. Extensive experiments conducted on diverse datasets underscore the effectiveness of Attention Mamba in extracting nonlinear dependencies and enhancing receptive fields, establishing superior performance among leading counterparts. Our codes will be available on GitHub.

Attention Mamba: Time Series Modeling with Adaptive Pooling Acceleration and Receptive Field Enhancements

TL;DR

Attention Mamba tackles the limitations of attention and receptive field constraints in time series forecasting by integrating an Adaptive Pooling block that accelerates attention while injecting global information and a bidirectional Mamba block to enhance long-short dependencies. The framework yields improved nonlinear dependency extraction and expanded receptive fields, leading to state-of-the-art performance across diverse real-world datasets. Extensive experiments and ranking analyses demonstrate robust advantages over Transformer-based, Linear-based, and other Mamba variants, with a favorable accuracy-to-complexity trade-off. The work offers practical guidance for efficient, nonlinear, long-range dependency modeling in time series and provides code on GitHub.

Abstract

"This work has been submitted to the lEEE for possible publication. Copyright may be transferred without noticeafter which this version may no longer be accessible." Time series modeling serves as the cornerstone of real-world applications, such as weather forecasting and transportation management. Recently, Mamba has become a promising model that combines near-linear computational complexity with high prediction accuracy in time series modeling, while facing challenges such as insufficient modeling of nonlinear dependencies in attention and restricted receptive fields caused by convolutions. To overcome these limitations, this paper introduces an innovative framework, Attention Mamba, featuring a novel Adaptive Pooling block that accelerates attention computation and incorporates global information, effectively overcoming the constraints of limited receptive fields. Furthermore, Attention Mamba integrates a bidirectional Mamba block, efficiently capturing long-short features and transforming inputs into the Value representations for attention mechanisms. Extensive experiments conducted on diverse datasets underscore the effectiveness of Attention Mamba in extracting nonlinear dependencies and enhancing receptive fields, establishing superior performance among leading counterparts. Our codes will be available on GitHub.

Paper Structure

This paper contains 12 sections, 5 equations, 2 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. METHODOLOGY
  3. Adaptive Pooling Acceleration
  4. Attention Mechanism with Bidirectional Mamba
  5. EXPERIMENTS
  6. Experimental Setup
  7. RESULTS AND PERFORMANCE ANALYSIS
  8. Nonlinear Dependency Extraction Effectiveness
  9. Receptive Field Enhancements
  10. Comprehensive Ranking Performance Analysis
  11. Memory and Complexity
  12. CONCLUSIONS

Figures (2)

  • Figure 1: The overall and detailed component architectures involved in Attention Mamba; (a) the overall architecture of Attention Mamba with paired RevIN layers used to reduce non-stationary occurrence; (b) the Adaptive Pooling block that accelerates attention computation and provides wider receptive fields; (c) the framework of Mamba that composes the core of the bidirectional Mamba block.
  • Figure 2: A snapshot of the attention map of PEMS04 with the forecasting window at 48 to visually demonstrate the enhanced receptive fields. Attention represents the input weighted by the Adaptive Pooling block, and W/O Attention represents the original input.