A Study of Posterior Stability for Time-Series Latent Diffusion

TL;DR

This paper shows that posterior collapse will reduce latent diffusion to a variational autoencoder (VAE), making it less expressive, and introduces a principled method: dependency measure, that quantifies the sensitivity of a recurrent decoder to input variables.

Abstract

Latent diffusion has demonstrated promising results in image generation and permits efficient sampling. However, this framework might suffer from the problem of posterior collapse when applied to time series. In this paper, we first show that posterior collapse will reduce latent diffusion to a variational autoencoder (VAE), making it less expressive. This highlights the importance of addressing this issue. We then introduce a principled method: dependency measure, that quantifies the sensitivity of a recurrent decoder to input variables. Using this tool, we confirm that posterior collapse significantly affects time-series latent diffusion on real datasets, and a phenomenon termed dependency illusion is also discovered in the case of shuffled time series. Finally, building on our theoretical and empirical studies, we introduce a new framework that extends latent diffusion and has a stable posterior. Extensive experiments on multiple real time-series datasets show that our new framework is free from posterior collapse and significantly outperforms previous baselines in time series synthesis.

Figures (5)

• Figure 1: The global and local dependency measures $m_{t, 0}, m_{t, t-1}$ (as defined in Sec. \ref{['sec:dep measure']}) respectively quantify the impacts of latent variable $\mathbf{z}$ and observation $\mathbf{x}_{t-1}$ on predicting the next one $\mathbf{x}_t$. We can see that the latent variable $\mathbf{z}$ of latent diffusion loses control over the condition generation $p^{\mathrm{gen}}(\mathbf{X} \mid \mathbf{z})$, with dependency illusion (as introduced in Sec. \ref{['sec:empirical studies']}) in the case of shuffled time series. In contrast, our framework has no such symptoms of posterior collapse.
• Figure 2: Dependency measures $m_{t, 0}, m_{t, t-1}$ averaged over $500$ multivariate time series, with $3$ standard deviations as the error bars. We can see that the latent variable $\mathbf{z}$ of latent diffusion has a vanishing impact on the decoder $\mathbf{f}^{\mathrm{dec}}$, a typical symptom of posterior collapse. We also observe a phenomenon of dependency illusion in the case of shuffled time series.
• Figure 3: In this example, path $\mathbf{X} \rightarrow \mathbf{z}^1$ is the variational inference (which gets rid of KL-divergence regularization) and path $\mathbf{X} \rightarrow \mathbf{z}^3$ shows the collapse simulation (which is to increase the sensitivity of decoder $\mathbf{f}^{\mathrm{dec}}$ to latent variable $\mathbf{z}$). Compared with time-series latent diffusion, our framework is free from posterior collapse and has a unlimited prior $p^{\mathrm{prior}}(\mathbf{z})$.
• Figure 4: The results of averaged dependency measures and error bars for our framework, which should be compared with those (e.g., Fig. \ref{['fig:demo case']}) of latent diffusion, showing that our framework has a stable posterior and is without dependency illusion.
• Figure : Training

Theorems & Definitions (5)

• Proposition 3.1: Gaussian Latent Variables
• proof
• Definition 3.2: Dependency Measures
• Proposition 3.3: Signed and Normalization Properties
• proof