Representation Learning of Multivariate Time Series using Attention and Adversarial Training

TL;DR

This work tackles robust representation learning for multivariate time series under data scarcity by combining a Transformer-based autoencoder with adversarial training to generate plausible synthetic sequences. The approach maps a prior $p_{noise}$ through a generator $G$ to produce artificial sequences, aiming to align the learned latent space with the data distribution $p_{data}$ via an adversarial discriminator. Evaluations using DTW, multivariate entropy, and t-SNE reveal that the Transformer with a Wasserstein GAN (TAE-WGAN) achieves the best dataset-matching similarity, though mode coverage and diversity remain challenging for some variants. Overall, the method shows promise for data augmentation and counterfactual generation in multivariate time series, with code and experiments available for reproducibility.

Abstract

A critical factor in trustworthy machine learning is to develop robust representations of the training data. Only under this guarantee methods are legitimate to artificially generate data, for example, to counteract imbalanced datasets or provide counterfactual explanations for blackbox decision-making systems. In recent years, Generative Adversarial Networks (GANs) have shown considerable results in forming stable representations and generating realistic data. While many applications focus on generating image data, less effort has been made in generating time series data, especially multivariate signals. In this work, a Transformer-based autoencoder is proposed that is regularized using an adversarial training scheme to generate artificial multivariate time series signals. The representation is evaluated using t-SNE visualizations, Dynamic Time Warping (DTW) and Entropy scores. Our results indicate that the generated signals exhibit higher similarity to an exemplary dataset than using a convolutional network approach.

Figures (5)

• Figure 1: The model augments the Transformer architecture with an autoencoder and an adversarial training scheme. For the Transformer decoder, the input time series is shifted to the right to serve as the output signal that is to be predicted. A memory vector is drawn from a prior distribution and is decoded to generate an artificial time series. The discriminator decides if a time series is true (from the dataset) or fake (artificially generated).
• Figure 2: t-SNE visualization of the NATOPS dataset. The colors represent the different classes (gestures).
• Figure 3: t-SNE representation of all models. The blue dots refer to the true time series from the validation dataset, while the orange dots refer to artificially generated time series from the models. Models without GAN regularization (TAE, CAE) did not learn the representation. TAE-GAN seems to have suffered from mode dropping and mode collapse.
• Figure 4: Ten random examples of time series signals from the NATOPS validation set.
• Figure 5: Ten different time series samples generated by TAE-WGAN. They show similar patterns to the validation dataset, however contain small fluctuations.