Heteroscedastic Temporal Variational Autoencoder For Irregular Time Series

TL;DR

The results show that the proposed architecture is better able to reflect variable uncertainty through time due to sparse and irregular sampling than a range of baseline and traditional models, as well as recent deep latent variable models that use homoscedastic output layers.

Abstract

Irregularly sampled time series commonly occur in several domains where they present a significant challenge to standard deep learning models. In this paper, we propose a new deep learning framework for probabilistic interpolation of irregularly sampled time series that we call the Heteroscedastic Temporal Variational Autoencoder (HeTVAE). HeTVAE includes a novel input layer to encode information about input observation sparsity, a temporal VAE architecture to propagate uncertainty due to input sparsity, and a heteroscedastic output layer to enable variable uncertainty in output interpolations. Our results show that the proposed architecture is better able to reflect variable uncertainty through time due to sparse and irregular sampling than a range of baseline and traditional models, as well as recently proposed deep latent variable models that use homoscedastic output layers.

Figures (5)

• Figure 1: (a) Architecture of UnTAN module. This module takes D-dimensional irregularly sampled time points $\mathbf{t} = [\mathbf{t}_{1} ,\cdots, \mathbf{t}_{D}]$ and corresponding observations $\mathbf{x} = [\mathbf{x}_{1} ,\cdots, \mathbf{x}_{D}]$ as keys and values and produces a fixed dimensional representation at the query time points $\mathbf{r} = [r_1, \cdots, r_K]$. Shared time embedding and attention function provide input to parallel intensity (INT) and value (VAL) encoding networks, whose outputs are subsequently fused via concatenation and an additional linear encoding layer. (b) Architecture of HeTVAE consisting of the UnTAND module to represent input uncertainty, parallel probabilistic (Prob) and deterministic (Det) encoding paths, and a heteroscedastic output layer that aims to reflect uncertainty due to input sparsity in the output distribution.
• Figure 2: We show example interpolations on the synthetic dataset. The set of 3 columns correspond to interpolation results with increasing numbers of observed points: 3, 10 and 20 respectively. The first, second and third rows correspond to STGP, HeTVAE and HTVAE mTAN respectively. The shaded region corresponds to $\pm$ one standard deviation. STGP and HetVAE exhibit variable output uncertainty in response to input sparsity while mTAN does not.
• Figure 3: In this figure, we show example interpolations of one dimension corresponding to Heart Rate on the PhysioNet dataset. The columns correspond to different examples. The rows correspond to STGP, HeTVAE, HTVAE mTAN, HeTVAE-DET-ALO and HeTVAE-DET respectively. The shaded region corresponds to $\pm$ one standard deviation. STGP, HeTVAE and HeTVAE-DET exhibit variable output uncertainty and good fit while mTAN and HETVAE-DET-ALO does not.
• Figure 4: Additional interpolation results on the synthetic dataset. The 3 columns correspond to interpolation results with increasing numbers of observed points: 3, 10 and 20 respectively. The shaded region corresponds to $\pm$ one standard deviation. STGP and HeTVAE exhibit variable output uncertainty in response to input sparsity while mTAN and HeTVAE - INT do not.
• Figure 5: In this figure, we show example interpolations on the synthetic dataset with increasing maximum inter-observation gap. The columns correspond to an inter-observation gap of size $20\%, 40\%, 60\%$ and $80\%$ of the length of original time series. The rows correspond to STGP, HeTVAE, HTVAE mTAN and HeTVAE-INT respectively. The shaded region corresponds to the confidence region. STGP and HeTVAE exhibit variable output uncertainty while mTAN and HeTVAE-INT does not.