Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learning

TL;DR

Frequency-masked Embedding Inference (FEI) is introduced, a novel non-contrastive method that completely eliminates the need for positive and negative samples and enables continuous semantic relationship modeling for time series.

Abstract

Contrastive learning underpins most current self-supervised time series representation methods. The strategy for constructing positive and negative sample pairs significantly affects the final representation quality. However, due to the continuous nature of time series semantics, the modeling approach of contrastive learning struggles to accommodate the characteristics of time series data. This results in issues such as difficulties in constructing hard negative samples and the potential introduction of inappropriate biases during positive sample construction. Although some recent works have developed several scientific strategies for constructing positive and negative sample pairs with improved effectiveness, they remain constrained by the contrastive learning framework. To fundamentally overcome the limitations of contrastive learning, this paper introduces Frequency-masked Embedding Inference (FEI), a novel non-contrastive method that completely eliminates the need for positive and negative samples. The proposed FEI constructs 2 inference branches based on a prompting strategy: 1) Using frequency masking as prompts to infer the embedding representation of the target series with missing frequency bands in the embedding space, and 2) Using the target series as prompts to infer its frequency masking embedding. In this way, FEI enables continuous semantic relationship modeling for time series. Experiments on 8 widely used time series datasets for classification and regression tasks, using linear evaluation and end-to-end fine-tuning, show that FEI significantly outperforms existing contrastive-based methods in terms of generalization. This study provides new insights into self-supervised representation learning for time series. The code is available at https://github.com/USTBInnovationPark/Frequency-masked-Embedding-Inference.

Figures (7)

• Figure 1: The overall structure of the proposed FEI. The original time series is fed into the encoder $f_\theta$ and a subspace projector $s_\phi$ to generate the original embedding. The target time series is constructed by applying random frequency masking, which is then fed into the momentum encoder—a smoothed copy of the original encoder updated via exponential moving average—to produce the target embedding. The goal of FEI is to enable the encoder to generate high-quality representation embeddings that can accurately infer the target embedding despite the presence of randomly masked frequency components(red dashed branch). Additionally, the representation embedding should also be capable of inferring mask embeddings by leveraging the differences between the target series and the original series (green dashed branch).
• Figure 2: Comparison of training loss curves for the first 20 epochs $w/$ and $w/o$ mask inference.
• Figure 3: Visualization of embedding inference results on Gesture dataset. The left side shows the inference results, and the right side shows the masks used to construct the target series.
• Figure 4: Accuracy results of FEI on the FD-B dataset under different masking ratios. The random masking ratio generated during each FEI training session lies between $\beta_1$ and $\beta_2$, as described in Section Frequency Masking.
• Figure 5: The linear evaluation results for varying momentum factors $\alpha$ on FD-B dataset.