A Survey on Time-Series Pre-Trained Models

TL;DR

The paper surveys Time-Series Pre-Trained Models (TS-PTMs), classifying them into supervised, unsupervised, and self-supervised categories and detailing their taxonomies, mechanisms, and downstream applicability. It provides a rigorous, uniform experimental comparison across 27 methods, 434 datasets, and 679 transfer-learning scenarios, highlighting that supervision-heavy transfers can underperform on small datasets while self-supervised and patch-based strategies excel in forecasting and anomaly detection. Core findings emphasize the strength of consistency-based methods (e.g., TS2Vec) and LLM-informed approaches (e.g., GPT4TS) under appropriate patching and Transformer architectures, as well as the need for large-scale time-series data akin to ImageNet for CV. The work offers practical guidance for selecting TS-PTMs based on downstream task and dataset characteristics and outlines future directions including large-scale datasets, LLMs, and robust training under label noise and adversarial settings.

Abstract

Time-Series Mining (TSM) is an important research area since it shows great potential in practical applications. Deep learning models that rely on massive labeled data have been utilized for TSM successfully. However, constructing a large-scale well-labeled dataset is difficult due to data annotation costs. Recently, pre-trained models have gradually attracted attention in the time series domain due to their remarkable performance in computer vision and natural language processing. In this survey, we provide a comprehensive review of Time-Series Pre-Trained Models (TS-PTMs), aiming to guide the understanding, applying, and studying TS-PTMs. Specifically, we first briefly introduce the typical deep learning models employed in TSM. Then, we give an overview of TS-PTMs according to the pre-training techniques. The main categories we explore include supervised, unsupervised, and self-supervised TS-PTMs. Further, extensive experiments involving 27 methods, 434 datasets, and 679 transfer learning scenarios are conducted to analyze the advantages and disadvantages of transfer learning strategies, Transformer-based models, and representative TS-PTMs. Finally, we point out some potential directions of TS-PTMs for future work.

Figures (15)

• Figure 1: Pre-training techniques for time series.
• Figure 2: Deep learning models used for time-series mining.
• Figure 3: The taxonomy of Pre-Trained Models for time-series mining.
• Figure 4: Universal encoder aims to learn general time-series representations through pre-training on various source datasets. The universal encoder is then fine-tuned on the target dataset for downstream TSM tasks.
• Figure 5: Aligned encoder aims to learn domain-invariant representations.