Generative Models for Long Time Series: Approximately Equivariant Recurrent Network Structures for an Adjusted Training Scheme

TL;DR

This work addresses the challenge of generative modeling for long time-series by introducing RVAE-ST, a translation-equivariant recurrent variational autoencoder with a fixed parameter budget. A repeat-vector latent is shared across time, and a progressive training curriculum gradually increases sequence length from $l=100$ toward longer horizons, enabling stable learning of long-range dependencies. Empirical results across five datasets show RVAE-ST achieving state-of-the-art or competitive performance on highly stationary data (as measured by Context-FID, ELBO, and discriminative scores) while remaining robust on less stationary sequences, highlighting the practical value of inductive biases for long sequence synthesis. The approach offers a simple, scalable alternative to more resource-intensive architectures and suggests avenues for extending the framework to diffusion-based methods and curriculum optimization.

Abstract

We present a simple yet effective generative model for time series data based on a Variational Autoencoder (VAE) with recurrent layers, referred to as the Recurrent Variational Autoencoder with Subsequent Training (RVAE-ST). Our method introduces an adapted training scheme that progressively increases the sequence length, addressing the challenge recurrent layers typically face when modeling long sequences. By leveraging the recurrent architecture, the model maintains a constant number of parameters regardless of sequence length. This design encourages approximate time-shift equivariance and enables efficient modeling of long-range temporal dependencies. Rather than introducing a fundamentally new architecture, we show that a carefully composed combination of known components can match or outperform state-of-the-art generative models on several benchmark datasets. Our model performs particularly well on time series that exhibit quasi-periodic structure,while remaining competitive on datasets with more irregular or partially non-stationary behavior. We evaluate its performance using ELBO, Fréchet Distance, discriminative scores, and visualizations of the learned embeddings.

Figures (19)

• Figure 1: This figure shows three excerpts from samples of the electric motor dataset (\ref{['sec:datasets']}), each with a sequence length of l = 1000. Sample (a) is taken from the original dataset. Sample (b) is generated using Diffusion-TS yuan2024diffusionts, a transformer-based state-of-the-art approach in time series generation. Sample (c) is generated using our model, trained with the proposed subsequent training scheme. The first row in the figure displays the voltage of one of the phases. In the original sample (a), the extremities of the voltage waveform exhibit pronounced volatility, particularly at peak and trough points. This characteristic remains clearly visible in the output of model (c), whereas it is notably reduced in model (b). The second row shows the DC-bus voltage. The signal is characterized by a distinctive sawtooth-like pattern, where three gradual drops are each followed by an abrupt upward jump. Model (c) reproduces this pattern well, although the waveform appears slightly smoothed compared to the original. Model (b) captures the general frequency of the signal but fails to replicate the sawtooth-like structure. The third row shows the effective motor current in the fixed coordinates of the stator. This channel exhibits both a high-frequency component, which gives the signal a noisy appearance, and a low-frequency oscillation reflecting the long-term behavior. Model (c) closely resembles the original (a), capturing both components. Model (b) approximates the low-frequency trend but deviates significantly in the high-frequency range.
• Figure 2: This figure illustrates the architecture of our model. Both the encoder and decoder are based on stacked LSTM layers. The encoder's final hidden states, denoted as $h_n$, are used to compute the parameters $\mu$ and $\log(\sigma)$, from which the latent variable $z$ is sampled. The latent variable $z$ is then repeated across all time steps and used as the input to the decoder. The decoder generates the sequence step-by-step, with each individual output passed through a time-distributed linear layer. This time-distributed layer applies the same linear transformation at each time step to the LSTM states, ensuring parameter sharing across the entire sequence during this transformation. Throughout the network, approximate equivariance is maintained with respect to time translation, and the number of trainable parameters remains constant regardless of the sequence length.
• Figure 3: Representative samples for each model at sequence length $l = 1000$ on a stationary dataset. RVAE-ST is the only model capable of consistently generating correct sinusoidal curves, demonstrating its ability to capture the stationary nature of the data.
• Figure 4: PCA plots for the EM and ECG datasets at sequence lengths of l = 100 and l = 1000. For the EM dataset, at l = 100, no significant differences are observed in the distributions of the models, with Time-Transformer showing a slightly less pronounced overlap. At l = 1000, the circular pattern of the data becomes more apparent, with RVAE-ST demonstrating the best performance, closely followed by Diffusion-TS. WaveGAN and TimeVAE show a few outliers and deviations, while TimeGAN exhibits almost no overlap. For the ECG dataset, at l = 100, RVAE-ST, TimeVAE, and Diffusion-TS show strong overlap with the original data, while WaveGAN and Time-Transformer exhibit less overlap, and TimeGAN shows almost no overlap. At l = 1000, RVAE-ST performs best, followed by WaveGAN and TimeVAE, with Diffusion-TS performing worse and TimeGAN and Time-Transformer showing minimal variability and no significant overlap.
• Figure 5: Example of a generated time series sample of length $l=5000$ from the Electric Motor dataset. The model was trained on sequences up to $l=1000$. The main characteristics of the dataset continue to be well synthesized in the extended sample. During generation, the model reaches a stable state in which the output patterns kind of repeat. As a result, slower trends, especially visible in the leff motorcurr stator fxd coord channel, are not fully reflected in the synthesis.