Time Series Representation Models

TL;DR

This work introduces Time Series Representation Models (TSRMs), a self-supervised, two-phase framework to learn category-specific representations for time series and to efficiently adapt them to forecasting and imputation tasks. The architecture combines a BERT-like encoder with an Encoding Layer, Representation Layer, and Attention-map Classifier, supported by three pretraining tasks—reconstruction, imputation, and binary category classification—and a dependent multi-task loss that balances representation, imputation, and classification objectives. Empirically, TSRMs achieve substantial improvements in imputation (up to 90% MAE reduction and notable RMSE gains) and forecasting (large MSE improvements on Electricity and Traffic) while dramatically reducing trainable parameters, though challenges remain with temporal embeddings on certain datasets. The work demonstrates a promising direction for resource-efficient, explainable time series analysis, with future work aimed at extending temporal embeddings, broader domain validation, and enhanced explainability evaluation.

Abstract

Time series analysis remains a major challenge due to its sparse characteristics, high dimensionality, and inconsistent data quality. Recent advancements in transformer-based techniques have enhanced capabilities in forecasting and imputation; however, these methods are still resource-heavy, lack adaptability, and face difficulties in integrating both local and global attributes of time series. To tackle these challenges, we propose a new architectural concept for time series analysis based on introspection. Central to this concept is the self-supervised pretraining of Time Series Representation Models (TSRMs), which once learned can be easily tailored and fine-tuned for specific tasks, such as forecasting and imputation, in an automated and resource-efficient manner. Our architecture is equipped with a flexible and hierarchical representation learning process, which is robust against missing data and outliers. It can capture and learn both local and global features of the structure, semantics, and crucial patterns of a given time series category, such as heart rate data. Our learned time series representation models can be efficiently adapted to a specific task, such as forecasting or imputation, without manual intervention. Furthermore, our architecture's design supports explainability by highlighting the significance of each input value for the task at hand. Our empirical study using four benchmark datasets shows that, compared to investigated state-of-the-art baseline methods, our architecture improves imputation and forecasting errors by up to 90.34% and 71.54%, respectively, while reducing the required trainable parameters by up to 92.43%. The source code is available at https://github.com/RobertLeppich/TSRM.

Figures (7)

• Figure 1: Illustration of the proposed Time Series Representation Models (TSRM) framework, primarily composed of $N$ encoding layers (ELs) (upper section in blue), accompanied by the representation layer (RL) (left, in green), and the attention map classifier (AC) (right, in red).
• Figure 2: Illustration of the artificial constructed imputation task during the pretraining step.
• Figure 3: Result of a fine-tuned forecasting model on the Traffic dataset with attention weights.
• Figure 4: Result of a fine-tuned forecasting model on the Traffic dataset with highlighted attention weights for all 5 ELs, starting with the first EL at the top and concluding with the last at the bottom
• Figure 5: Illustration of the setup of a forecasting task during fine-tuning.