T-Rep: Representation Learning for Time Series using Time-Embeddings

TL;DR

T-Rep presents a self-supervised framework for learning timestep-level representations of multivariate time series by jointly learning learnable time-embeddings with a convolutional encoder. Two pretext tasks—Time-embedding Divergence Prediction and Time-embedding-conditioned Forecasting—drive the model to capture trend, periodicity, and distribution shifts while enforcing contextual consistency and enabling a hierarchical, multiscale representation. Empirically, T-Rep outperforms state-of-the-art SSL methods across anomaly detection, classification, and forecasting, and shows superior robustness to missing data with a compact latent space, while offering interpretable temporal trajectories. The approach offers a principled way to encode temporal structure in SSL for time series, with potential impact in medicine, climate, and sensor-domain applications where data are noisy, incomplete, and unlabeled.

Abstract

Multivariate time series present challenges to standard machine learning techniques, as they are often unlabeled, high dimensional, noisy, and contain missing data. To address this, we propose T-Rep, a self-supervised method to learn time series representations at a timestep granularity. T-Rep learns vector embeddings of time alongside its feature extractor, to extract temporal features such as trend, periodicity, or distribution shifts from the signal. These time-embeddings are leveraged in pretext tasks, to incorporate smooth and fine-grained temporal dependencies in the representations, as well as reinforce robustness to missing data. We evaluate T-Rep on downstream classification, forecasting, and anomaly detection tasks. It is compared to existing self-supervised algorithms for time series, which it outperforms in all three tasks. We test T-Rep in missing data regimes, where it proves more resilient than its counterparts. Finally, we provide latent space visualisation experiments, highlighting the interpretability of the learned representations.

Figures (5)

• Figure 1: T-Rep architecture and workflow. The left part shows how the different modules interact (linear projection, time-embedding module, and encoder). The middle part shows the 4 pretext tasks used to train the model, and the kind of features they extract. The right hand side shows the loss computation: a linear combination of individual pretext task losses is passed to the hierarchical loss algorithm (see Appendix \ref{['pretext_task_weights']}).
• Figure 2: Illustration of T-Rep's robustness to missing data on UCR archive datasets. (a) shows heatmap representations of T-Rep and TS2Vec when faced with missing data, and (b) shows accuracy against percentage of missing data in a classification task for T-Rep, TS2Vec ts2vec and T-Loss t_loss. Error bars denote the standard deviation over 6 train-test runs.
• Figure 3: Time series representation trajectories in the latent space. The top row shows the input signal, and the bottom row shows a heatmap of the representations through time. Only the 15 most varying dimensions are shown for each representation. Data for the two left-most figures comes from UCR archive datasets, and the righ-most figure's data is synthetic.
• Figure 4: UMAP visualisations of the learned representations of T-Rep and TS2Vec on 5 UCR datasets. Each color represents a different class. TS2Vec results are produced using the official code ts2vec_repo.
• Figure 5: Hierarchical loss algorithm ts2vec