Towards Generalisable Time Series Understanding Across Domains

TL;DR

OTiS addresses the challenge of generalising time series understanding across heterogeneous domains by introducing a multi-domain pre-training framework that combines a domain-specific tokeniser, a dual masking strategy, and a normalised cross-correlation loss. Pre-trained on a large, diverse corpus spanning $8$ domains with $640{,}187$ samples and $11{,}052{,}756{,}981$ time points, OTiS achieves competitive performance across classification, regression, and forecasting tasks, and demonstrates zero-shot capabilities and domain-adaptation via learnable domain signatures. Analyses of domain signatures reveal that the model captures inter-variate relationships and temporal patterns consistent with domain-specific structure (e.g., EEG/ECG electrode layouts and climatological relationships), enabling transfer to unseen domains with limited data. Together, these results establish a foundation for general time series analysis with potential impact across medicine, engineering, natural sciences, and finance, while acknowledging limitations related to data curation and the need for even larger pre-training corpora for further gains.

Abstract

Recent breakthroughs in natural language processing and computer vision, driven by efficient pre-training on large datasets, have enabled foundation models to excel on a wide range of tasks. However, this potential has not yet been fully realised in time series analysis, as existing methods fail to address the heterogeneity in large time series corpora. Prevalent in domains ranging from medicine to finance, time series vary substantially in characteristics such as variate count, inter-variate relationships, temporal patterns, and sampling frequency. To address this, we introduce a novel pre-training paradigm specifically designed to handle time series heterogeneity. We propose a tokeniser with learnable domain signatures, a dual masking strategy, and a normalised cross-correlation loss, enabling our open model for general time series analysis (OTiS) to efficiently learn from large time series corpora. Extensive benchmarking on diverse tasks, such as classification, regression, and forecasting, demonstrates that OTiS outperforms state-of-the-art baselines. Our code and pre-trained weights are available at https://github.com/oetu/otis.

Figures (15)

• Figure 1: Open model for general time series analysis (OTiS). Its tokeniser accounts for varying time series characteristics across domains, such as distinct variate counts, inter-variate relationships, temporal patterns, and sampling frequencies. It can be fine-tuned on limited data from any domain, including previously unseen ones, to perform classification, regression, and forecasting tasks.
• Figure 2: Pre-training of OTiS. A time series is split into fixed-size patches, which are then embedded using a universal patch projector. In addition to a temporal embedding, each patch embedding is modulated with a domain-specific variate embedding to account for the unique characteristics of a domain. The resulting input tokens are masked with a dual masking strategy and reconstructed to optimise OTiS. The reconstruction is guided by a combination of mean squared error (MSE) and normalised cross-correlation (NCC) loss terms.
• Figure 3: First two principal components of zero-shot sinusoidal representations extracted by OTiS-Base. We freeze OTiS after pre-training and use randomly initialised variate embeddings. The output features of the encoder are averaged to obtain a global representation. OTiS has an intrinsic understanding of fundamental time series properties, such as (a) frequency, (b) amplitude, (c) offset, and (d) phase. It effectively disentangles (e) simultaneous variations of these properties, providing a strong foundation for general time series analysis.
• Figure 4: Scaling study. Shaded regions indicate the standard deviation across $5$ seeds. Downstream performance across tasks generally scales with dataset size. Scaling the model size requires even larger pre-training corpora to be effective.
• Figure 5: Ablation study. Downstream performance is analysed across $5$ seeds. A leave-one-out approach is used to evaluate the influence of each component. The default setup, incorporating all components, achieves superior performance across tasks.