CMamba: Channel Correlation Enhanced State Space Models for Multivariate Time Series Forecasting

TL;DR

A refined Mamba variant tailored for time series forecasting is introduced that incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data-dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mixup mechanism to mitigate overfitting.

Abstract

Recent advancements in multivariate time series forecasting have been propelled by Linear-based, Transformer-based, and Convolution-based models, with Transformer-based architectures gaining prominence for their efficacy in temporal and cross-channel mixing. More recently, Mamba, a state space model, has emerged with robust sequence and feature mixing capabilities. However, the suitability of the vanilla Mamba design for time series forecasting remains an open question, particularly due to its inadequate handling of cross-channel dependencies. Capturing cross-channel dependencies is critical in enhancing the performance of multivariate time series prediction. Recent findings show that self-attention excels in capturing cross-channel dependencies, whereas other simpler mechanisms, such as MLP, may degrade model performance. This is counterintuitive, as MLP, being a learnable architecture, should theoretically capture both correlations and irrelevances, potentially leading to neutral or improved performance. Diving into the self-attention mechanism, we attribute the observed degradation in MLP performance to its lack of data dependence and global receptive field, which result in MLP's lack of generalization ability. Based on the above insights, we introduce a refined Mamba variant tailored for time series forecasting. Our proposed model, \textbf{CMamba}, incorporates a modified Mamba (M-Mamba) module for temporal dependencies modeling, a global data-dependent MLP (GDD-MLP) to effectively capture cross-channel dependencies, and a Channel Mixup mechanism to mitigate overfitting. Comprehensive experiments conducted on seven real-world datasets demonstrate the efficacy of our model in improving forecasting performance.

Figures (20)

• Figure 1: An illustration of the relationship of variables in the ETT dataset. HULL means High UseLess Load and MULL means Middle UseLess Load.
• Figure 2: A case study on the Electricity dataset using different modules to model cross-channel dependencies. The look-back length is 96. Avg means the average metrics for four horizons.
• Figure 3: The overall framework of CMamba. (a) The Channel Mixup module, active during training, fuses different channels of a sample to create a virtual sample. New samples are normalized via instance norm and segmented into patches before being fed into the model. (b) The CMamba block consists of two parts: the M-Mamba module and GDD-MLP before residual connection. (c) The M-Mamba module captures cross-time dependencies. (d) The GDD-MLP module captures cross-channel dependencies.
• Figure 4: Architectures of the vanilla Mamba module and M-Mamba module.
• Figure 5: Loss curves for the Traffic dataset with look-back length and prediction length fixed at $96$. All curves combine the specified module with M-Mamba. For example, GDD-MLP + Channel Mixup corresponds to CMamba.