TimeDRL: Disentangled Representation Learning for Multivariate Time-Series

TL;DR

TimeDRL tackles the challenge of learning robust, label-efficient representations from multivariate time-series by disentangling timestamp-level and instance-level embeddings using patched inputs and a [CLS] token. It couples a non-augmented timestamp-predictive task with a dropout-based, negative-sample-free instance-contrastive task, enabling two complementary learning signals without introducing augmentation-induced inductive bias. Empirical results across 6 forecasting and 5 classification datasets show substantial gains in MSE and accuracy, with strong evidence of component necessity from comprehensive ablations and a demonstrated benefit in semi-supervised settings. The work advances time-series representation learning toward a practical, generalizable framework that reduces labeling needs while maintaining high downstream performance.

Abstract

Multivariate time-series data in numerous real-world applications (e.g., healthcare and industry) are informative but challenging due to the lack of labels and high dimensionality. Recent studies in self-supervised learning have shown their potential in learning rich representations without relying on labels, yet they fall short in learning disentangled embeddings and addressing issues of inductive bias (e.g., transformation-invariance). To tackle these challenges, we propose TimeDRL, a generic multivariate time-series representation learning framework with disentangled dual-level embeddings. TimeDRL is characterized by three novel features: (i) disentangled derivation of timestamp-level and instance-level embeddings from patched time-series data using a [CLS] token strategy; (ii) utilization of timestamp-predictive and instance-contrastive tasks for disentangled representation learning, with the former optimizing timestamp-level embeddings with predictive loss, and the latter optimizing instance-level embeddings with contrastive loss; and (iii) avoidance of augmentation methods to eliminate inductive biases, such as transformation-invariance from cropping and masking. Comprehensive experiments on 6 time-series forecasting datasets and 5 time-series classification datasets have shown that TimeDRL consistently surpasses existing representation learning approaches, achieving an average improvement of forecasting by 58.02% in MSE and classification by 1.48% in accuracy. Furthermore, extensive ablation studies confirmed the relative contribution of each component in TimeDRL's architecture, and semi-supervised learning evaluations demonstrated its effectiveness in real-world scenarios, even with limited labeled data. The code is available at https://github.com/blacksnail789521/TimeDRL.

Figures (6)

• Figure 1: Two categories of representation learning in the time-series domain. The top section (a) represents entangled representation learning, where timestamp-level embeddings are first derived and then a pooling method is applied to extract instance-level embeddings. The bottom section (b) illustrates disentangled representation learning, which involves deriving timestamp-level and instance-level embeddings in a disentangled manner. Our proposed TimeDRL employs disentangled derivation of timestamp-level and instance-level embeddings.
• Figure 2: Two categories of self-supervised learning. The left sections (a) and (c) represent predictive learning, utilizing a single representation to predict inherent data characteristics. The right sections (b) and (d) illustrate contrastive learning, emphasizing the distinction of subtle differences between data samples. Our proposed TimeDRL avoids using any augmentation methods in both categories to enable robust learning and prevent inductive bias ((c) and (d)).
• Figure 3: TimeDRL framework. The framework is illustrated in two sections: (a) Pre-training Stage and (b) Fine-tuning Stage. In the pre-training stage (a), we adopt a Siamese network architecture with a Transformer encoder as our encoder $f_\theta$ to generate two distinct views of embeddings from the same input. This is achieved by exploiting the inherent randomness of the dropout layers within the encoder, thus eliminating the need for data augmentations. The fine-tuning stage (b) demonstrates the application of these embeddings in downstream tasks, showcasing the adaptability of TimeDRL's pre-trained embeddings for time-series forecasting and classification.
• Figure 4: Comparison of training time (in seconds) in the pre-training stage on forecasting datasets.
• Figure 5: Semi-supervised learning. To simulate real-world scenarios with limited availability of labeled data, we experiment with different portions of the labels in our datasets. The top sections (a-c) represent time-series forecasting, while the bottom sections (d-f) correspond to time-series classification. 'TimeDRL (FT)' indicates that we fine-tune the encoder during the downstream tasks.