Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction

Wenbo Yan, Hanzhong Cao, Ying Tan

TL;DR

The paper targets robust time series forecasting with Mamba-based models by addressing their limited selectivity and noise suppression. It introduces Repetitive Contrastive Learning (RCL), which pretrains a single Mamba block using Repeating Sequence Augmentation and intra/inter-sequence contrast, then transfers the learned selectivity to backbone models. The approach yields state-of-the-art or competitive improvements across multiple datasets and backbones, with two new metrics (Focus Ratio and Memory Entropy) to quantify selectivity, and demonstrates negligible memory overhead. The findings suggest RCL is broadly applicable to Mamba-based architectures and can enhance forecasting by sharpening timestep attention and denoising capabilities, including under few-shot and missing-data conditions.

Abstract

Long sequence prediction is a key challenge in time series forecasting. While Mamba-based models have shown strong performance due to their sequence selection capabilities, they still struggle with insufficient focus on critical time steps and incomplete noise suppression, caused by limited selective abilities. To address this, we introduce Repetitive Contrastive Learning (RCL), a token-level contrastive pretraining framework aimed at enhancing Mamba's selective capabilities. RCL pretrains a single Mamba block to strengthen its selective abilities and then transfers these pretrained parameters to initialize Mamba blocks in various backbone models, improving their temporal prediction performance. RCL uses sequence augmentation with Gaussian noise and applies inter-sequence and intra-sequence contrastive learning to help the Mamba module prioritize information-rich time steps while ignoring noisy ones. Extensive experiments show that RCL consistently boosts the performance of backbone models, surpassing existing methods and achieving state-of-the-art results. Additionally, we propose two metrics to quantify Mamba's selective capabilities, providing theoretical, qualitative, and quantitative evidence for the improvements brought by RCL.

Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction

TL;DR

The paper targets robust time series forecasting with Mamba-based models by addressing their limited selectivity and noise suppression. It introduces Repetitive Contrastive Learning (RCL), which pretrains a single Mamba block using Repeating Sequence Augmentation and intra/inter-sequence contrast, then transfers the learned selectivity to backbone models. The approach yields state-of-the-art or competitive improvements across multiple datasets and backbones, with two new metrics (Focus Ratio and Memory Entropy) to quantify selectivity, and demonstrates negligible memory overhead. The findings suggest RCL is broadly applicable to Mamba-based architectures and can enhance forecasting by sharpening timestep attention and denoising capabilities, including under few-shot and missing-data conditions.

Abstract

Long sequence prediction is a key challenge in time series forecasting. While Mamba-based models have shown strong performance due to their sequence selection capabilities, they still struggle with insufficient focus on critical time steps and incomplete noise suppression, caused by limited selective abilities. To address this, we introduce Repetitive Contrastive Learning (RCL), a token-level contrastive pretraining framework aimed at enhancing Mamba's selective capabilities. RCL pretrains a single Mamba block to strengthen its selective abilities and then transfers these pretrained parameters to initialize Mamba blocks in various backbone models, improving their temporal prediction performance. RCL uses sequence augmentation with Gaussian noise and applies inter-sequence and intra-sequence contrastive learning to help the Mamba module prioritize information-rich time steps while ignoring noisy ones. Extensive experiments show that RCL consistently boosts the performance of backbone models, surpassing existing methods and achieving state-of-the-art results. Additionally, we propose two metrics to quantify Mamba's selective capabilities, providing theoretical, qualitative, and quantitative evidence for the improvements brought by RCL.

Paper Structure

This paper contains 47 sections, 24 equations, 10 figures, 17 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Multivariate Time Series Forecasting
  4. Mamba Block
  5. Definition of Mamba's Selectivity
  6. Definition of a Selectivity Measurement
  7. Method
  8. Repeating Sequence Augmentation
  9. Repetitive Contrastive Learning
  10. Replace and Freezing Method
  11. Experiment
  12. Basic Information
  13. Mamba-based Baseline
  14. Temporal Baseline
  15. Temporal Pre-training Baseline
  16. ...and 32 more sections

Figures (10)

  • Figure 1: Impact of Noise Sensitivity on Prediction Results
  • Figure 2: (a)The structure of the mamba block. (b)The Process of Sequence Modeling with Mamba
  • Figure 3: Process of the proposed method. Including Repeating Sequence Augmentation and Repetitive Contrastive Learning (RCL), with RCL consisting of Intra-sequence contrast and Inter-sequence contrast.
  • Figure 4: Visualization of Predictions when Replacing 100%/50% of the Layers and Frozen/Not Forzen Parameters
  • Figure 5: Hidden state and $\Delta$ corresponding to the input time series.
  • ...and 5 more figures