Large Pre-trained time series models for cross-domain Time series analysis tasks

TL;DR

This work addresses the challenge of building a unified pre-trained model for cross-domain time-series analysis. It introduces Large Pre-trained Time-series Models (LPTM) with an adaptive segmentation module that learns dataset-specific tokenization, enabling effective multi-domain self-supervised pre-training on $\mathcal{D}_{\text{pre}}$ and SSL tasks. LPTM achieves competitive zero-shot and superior fine-tuned performance across forecasting and classification benchmarks while using less pre-training data and training time than prior baselines. The approach demonstrates strong data and compute efficiency, with segments that adapt to domain-specific dynamics and high-variance regions, offering a practical path to cross-domain time-series foundation models. Potential extensions include calibrated uncertainty estimation and multimodal integration to broaden applicability in critical domains like public health and energy.

Abstract

Large pre-trained models have been vital in recent advancements in domains like language and vision, making model training for individual downstream tasks more efficient and provide superior performance. However, tackling time-series analysis tasks usually involves designing and training a separate model from scratch leveraging training data and domain expertise specific to the task. We tackle a significant challenge for pre-training a foundational time-series model from multi-domain time-series datasets: extracting semantically useful tokenized inputs to the model across heterogenous time-series from different domains. We propose Large Pre-trained Time-series Models (LPTM) that introduces a novel method of adaptive segmentation that automatically identifies optimal dataset-specific segmentation strategy during pre-training. This enables LPTM to perform similar to or better than domain-specific state-of-art model when fine-tuned to different downstream time-series analysis tasks and under zero-shot settings. LPTM achieves superior forecasting and time-series classification results taking up to 40% less data and 50% less training time compared to state-of-art baselines. Code: www.github.com/AdityaLab/Samay

Figures (5)

• Figure 1: Overview of LPTM. The input time-series $y^{(1\dots T)}$ is first segmented based on a scoring function optimized using SSL loss. The segments are fed as individual tokens to the transformer encoder to get output embeddings of time-series that are used for downstream tasks.
• Figure 2: Performance of LPTM and best baseline with varying fractions of training data. In most cases LPTM significantly outperforms baselines with lower amount of data.
• Figure 3: Segmentation learned by LPTM
• Figure 4: Performance of LPTM and best baseline with varying fractions of training data. In most cases LPTM significantly outperforms baselines with lower amount of data.
• Figure 5: Effect of $\gamma$ on performance(RMSE) for different benchmarks