TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling

TL;DR

TimeSiam addresses the challenge that traditional time-series pre-training methods either distort temporal structure via masking or miss fine-grained dynamics via instance-level contrastive objectives. It introduces Siamese encoders that learn correlations between randomly sampled past and current subseries, employing a mask-based past-to-current reconstruction objective and learnable lineage embeddings to distinguish temporal distances. Across 13 standard benchmarks for forecasting and classification, TimeSiam delivers state-of-the-art results in both in-domain and cross-domain settings, with gains amplified by large-scale pre-training on TSld (e.g., $1 ext{G}$ data). The approach demonstrates strong generalization, efficient representation learning, and practical potential for reducing labeling needs in real-world time-series tasks, while remaining simple and adaptable to extended inputs via lineage conditioning. Key mathematical elements include the reconstruction loss ${\cal L}_{\text{reconstruction}} = \|\mathbf{x}^{\text{curr}} - \widehat{\mathbf{x}}^{\text{curr}}\|_2^2$ and the lineage-enabled representation ${\overline{\mathbf{h}}_{e}}$ produced by averaging or concatenating lineage-conditioned encodings, ensuring diverse temporal semantics are captured across distanced subseries. $T$ denotes subseries length and $C$ the number of channels in each subseries.

Abstract

Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. Prior methods are mainly based on pre-training techniques well-acknowledged in vision or language, such as masked modeling and contrastive learning. However, randomly masking time series or calculating series-wise similarity will distort or neglect inherent temporal correlations crucial in time series data. To emphasize temporal correlation modeling, this paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks. Concretely, TimeSiam pre-trains Siamese encoders to capture intrinsic temporal correlations between randomly sampled past and current subseries. With a simple data augmentation method (e.g.~masking), TimeSiam can benefit from diverse augmented subseries and learn internal time-dependent representations through a past-to-current reconstruction. Moreover, learnable lineage embeddings are also introduced to distinguish temporal distance between sampled series and further foster the learning of diverse temporal correlations. TimeSiam consistently outperforms extensive advanced pre-training baselines, demonstrating superior forecasting and classification capabilities across 13 standard benchmarks in both intra- and cross-domain scenarios.

Figures (12)

• Figure 1: Comparison on time series pre-training frameworks. (a) Masked modeling: reconstruct the masked series. (b) Contrastive learning: Repulse different series (negative pairs) while attracting two augmentations from the same series (positive pairs). (c) TimeSiam: reconstruct masked current series $\mathbf{x}^{\text{curr}}$ from randomly sampled past observation $\mathbf{x}^{\text{past}}$.
• Figure 2: The overall design of TimeSiam, which establishes correlations between subseries randomly sampled from different timestamps using Siamese encoders. It integrates learnable lineage embeddings to enhance the capacity for temporal-related representation learning.
• Figure 3: Comparison of time series pre-training baselines for forecasting (MSE $\downarrow$) and classification (Accuracy $\uparrow$) tasks. This comparison included both contrastive-based and masking-based methods, covering both in- (left) and cross-domain (right) settings.
• Figure 4: Fine-tuning the pre-trained model to the inputs with extended length {96, 192, 288, 384, 576} based on iTransformer liu2023itransformer. The MSE averaged from all predicted horizons $\{96,192,336,720\}$ is reported. Additional results are in the Appendix \ref{['app:adapt_length_full_result']}.
• Figure 5: Linear probing on in-domain forecasting setting. Average results (MSE) are reported. Full results are shown in Table \ref{['tab:PatchTST_forecasting_linear_indomain_full']}.