Variational quantization for state space models

TL;DR

A new forecasting model is proposed that combines discrete state space hidden Markov models with recent neural network architectures and training procedures inspired by vector quantized variational autoencoders and outperforms other state-of-the-art solutions.

Abstract

Forecasting tasks using large datasets gathering thousands of heterogeneous time series is a crucial statistical problem in numerous sectors. The main challenge is to model a rich variety of time series, leverage any available external signals and provide sharp predictions with statistical guarantees. In this work, we propose a new forecasting model that combines discrete state space hidden Markov models with recent neural network architectures and training procedures inspired by vector quantized variational autoencoders. We introduce a variational discrete posterior distribution of the latent states given the observations and a two-stage training procedure to alternatively train the parameters of the latent states and of the emission distributions. By learning a collection of emission laws and temporarily activating them depending on the hidden process dynamics, the proposed method allows to explore large datasets and leverage available external signals. We assess the performance of the proposed method using several datasets and show that it outperforms other state-of-the-art solutions.

Figures (9)

• Figure 1: Illustration of the proposed framework with 3 hidden states. Given the past of a time series $y^i_{t-w+1:t}$ and possible additional external signals $w^i_{t-w+1:t}$ (called $z$ in the figure), a trajectory of the hidden state $\hat{x}^i_{t+1:t+h}$ is drawn using the law of the hidden states. Then, conditioned on the values taken by the hidden process, one of the emission law is activated and used to compute the final prediction $\hat{y}^i_{t+1:t+h}$.
• Figure 2: Predictions on the fashion dataset. (Top Left) Prediction of the two emission laws when the hidden state is 0 or 1. (Top Right) Empirical distributions of the hidden states. (Bottom) Simulated predictions with our model using external signals.
• Figure 3: Example of model architecture. Example of architecture used for the proposed approach on the Fashion dataset. (Left) Model used to compute parameters of the k-th emission law. (Middle) Model used to compute the hidden state probabilities. (Right) Model used to approximate the posterior law of the hidden states.
• Figure 4: Ours vs Ours-es predictions.Ours and Ours-es model predictions on three fashion time series: (Top) "br_female_shoes_262", (Middle) "eu_female_outerwear_177", (Bottom) "eu_female_texture_80". On several fashion time series, Ours-es correctly leverages the influencers external signal and capture sudden non-stationary evolution impossible to forecast without them.
• Figure 5: Proposed method with 4 hidden states predictions. Emission laws predictions of the proposed model with 4 hidden states on three fashion time series: (Top) "br_female_shoes_262", (Middle) "eu_female_outerwear_177", (Bottom) "eu_female_texture_80". For this model, the influencers external signal was only given to the third and the fourth emission laws. The third and the fourth emission laws learned different distribution but the first and the second ones seems to be redundant.