Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stepwise Variational Inference with Vine Copulas

Elisabeth Griesbauer, Leiv Rønneberg, Arnoldo Frigessi, Claudia Czado, Ingrid Hobæk Haff

Abstract

We propose stepwise variational inference (VI) with vine copulas: a universal VI procedure that combines vine copulas with a novel stepwise estimation procedure of the variational parameters. Vine copulas consist of a nested sequence of trees built from copulas, where more complex latent dependence can be modeled with increasing number of trees. We propose to estimate the vine copula approximate posterior in a stepwise fashion, tree by tree along the vine structure. Further, we show that the usual backward Kullback-Leibler divergence cannot recover the correct parameters in the vine copula model, thus the evidence lower bound is defined based on the Rényi divergence. Finally, an intuitive stopping criterion for adding further trees to the vine eliminates the need to pre-define a complexity parameter of the variational distribution, as required for most other approaches. Thus, our method interpolates between mean-field VI (MFVI) and full latent dependence. In many applications, in particular sparse Gaussian processes, our method is parsimonious with parameters, while outperforming MFVI.

Stepwise Variational Inference with Vine Copulas

Abstract

We propose stepwise variational inference (VI) with vine copulas: a universal VI procedure that combines vine copulas with a novel stepwise estimation procedure of the variational parameters. Vine copulas consist of a nested sequence of trees built from copulas, where more complex latent dependence can be modeled with increasing number of trees. We propose to estimate the vine copula approximate posterior in a stepwise fashion, tree by tree along the vine structure. Further, we show that the usual backward Kullback-Leibler divergence cannot recover the correct parameters in the vine copula model, thus the evidence lower bound is defined based on the Rényi divergence. Finally, an intuitive stopping criterion for adding further trees to the vine eliminates the need to pre-define a complexity parameter of the variational distribution, as required for most other approaches. Thus, our method interpolates between mean-field VI (MFVI) and full latent dependence. In many applications, in particular sparse Gaussian processes, our method is parsimonious with parameters, while outperforming MFVI.
Paper Structure (55 sections, 8 theorems, 114 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 55 sections, 8 theorems, 114 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contributions
  4. Background
  5. Variational Inference (VI)
  6. Rényi $\alpha$-divergence and VI
  7. Rényi $\alpha$-divergence
  8. Rényi divergence VI and VR-IWAE bound
  9. VR-IWAE bound and $\alpha$
  10. Reparametrization trick
  11. Vine Copulas
  12. Stepwise Variational Inference with Vine Copulas
  13. Theoretical justification for low $\alpha$
  14. VI with VR-IWAE bound and reparameterization
  15. Global stopping criterion
  16. ...and 40 more sections

Key Result

Theorem 3.1

The parameters $\bm{\phi} = (\boldsymbol{\nu},\mathbf{D}_{\Psi},\mathbf{R}_{\Psi})$ of $q(\mathbf{z}; \bm{\phi})$ obtained when minimizing the forward KL in the proposed stepwise manner are the true parameters, i.e $\boldsymbol{\nu}=\boldsymbol{\mu}$, $\mathbf{D}_{\Psi}=\mathbf{D}_{\Sigma}$ and $\ma

Figures (7)

  • Figure 1: Contour plots of samples from NUTS (black) regarded as ground truth and variational approximations obtained with stepwise VI with vines (blue), MFVI (orange) and GC-VI (green).
  • Figure 2: A D-vine tree sequence on 4 elements.
  • Figure 3: NLPD across a range of inducing point values for the pumadyn32nm dataset. We compare the mean-field SGPR (MF-SGPR), a full rank SGPR (FullRank-SGPR), and the vine at tree level $\tau=1$ to a full (non-sparse) GP fit (red dashed line).
  • Figure 4: Evolution of the corresponding correlation matrix associated with the variational matrix $S$ at different tree levels $t$ of vine copula approximate posterior, for the pumadyn32nm example using 50 inducing points.
  • Figure 5: A D-vine tree sequence on 4 elements.
  • ...and 2 more figures

Theorems & Definitions (28)

  • Definition 2.1: Truncation of the Vine Copula at Level $\tau$
  • Theorem 3.1
  • Theorem 3.2
  • Definition 3.1
  • Theorem 3.2: Sklar's Theorem
  • Definition 3.3: bivariate Gaussian copula
  • Definition 3.4
  • Example 3.5: Pair copula decomposition
  • Definition 3.6: (Regular) Vine tree sequence
  • Definition 3.7: D-vine
  • ...and 18 more