Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders

Rogelio A Mancisidor, Robert Jenssen, Shujian Yu, Michael Kampffmeyer

TL;DR

This research proposes a novel ELBO that approximates the joint likelihood of the multimodal data by learning the contribution of each subset of modalities and demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality.

Abstract

Multimodal learning with variational autoencoders (VAEs) requires estimating joint distributions to evaluate the evidence lower bound (ELBO). Current methods, the product and mixture of experts, aggregate single-modality distributions assuming independence for simplicity, which is an overoptimistic assumption. This research introduces a novel methodology for aggregating single-modality distributions by exploiting the principle of consensus of dependent experts (CoDE), which circumvents the aforementioned assumption. Utilizing the CoDE method, we propose a novel ELBO that approximates the joint likelihood of the multimodal data by learning the contribution of each subset of modalities. The resulting CoDE-VAE model demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality, as well as generating more precise log-likelihood estimations. CoDE-VAE further minimizes the generative quality gap as the number of modalities increases. In certain cases, it reaches a generative quality similar to that of unimodal VAEs, which is a desirable property that is lacking in most current methods. Finally, the classification accuracy achieved by CoDE-VAE is comparable to that of state-of-the-art multimodal VAE models.

Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoders

TL;DR

This research proposes a novel ELBO that approximates the joint likelihood of the multimodal data by learning the contribution of each subset of modalities and demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality.

Abstract

Multimodal learning with variational autoencoders (VAEs) requires estimating joint distributions to evaluate the evidence lower bound (ELBO). Current methods, the product and mixture of experts, aggregate single-modality distributions assuming independence for simplicity, which is an overoptimistic assumption. This research introduces a novel methodology for aggregating single-modality distributions by exploiting the principle of consensus of dependent experts (CoDE), which circumvents the aforementioned assumption. Utilizing the CoDE method, we propose a novel ELBO that approximates the joint likelihood of the multimodal data by learning the contribution of each subset of modalities. The resulting CoDE-VAE model demonstrates better performance in terms of balancing the trade-off between generative coherence and generative quality, as well as generating more precise log-likelihood estimations. CoDE-VAE further minimizes the generative quality gap as the number of modalities increases. In certain cases, it reaches a generative quality similar to that of unimodal VAEs, which is a desirable property that is lacking in most current methods. Finally, the classification accuracy achieved by CoDE-VAE is comparable to that of state-of-the-art multimodal VAE models.
Paper Structure (45 sections, 3 theorems, 25 equations, 21 figures, 15 tables, 1 algorithm)

This paper contains 45 sections, 3 theorems, 25 equations, 21 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Consensus of Experts with Stochastic Dependence
  5. Toy example:
  6. Dependency between expert distributions:
  7. Evidence Lower Bound
  8. Experiments
  9. MNIST-SVHN-Text
  10. PolyMNIST
  11. Generative quality gap:
  12. Log-likelihoods, generative quality, and coherence:
  13. CUB
  14. Contribution of each $\mathcal{L}_k$ to optimization
  15. Ablation Experiments: the effect of ${\bm{\pi}}$ and $\rho$
  16. ...and 30 more sections

Key Result

Lemma 1

Assume that the error of estimation $\bm{e}_k$ of the k-th consensus distribution is a random variable with multivariate Gaussian distribution $\bm{e}_k \sim \mathcal{N}(\bm{0}, \bm{\Sigma}_k)$, where $\bm{0}$ is a $[M^' \cdot D \times 1]$ vector, $M^'$ is the number of experts who are elements of t where and $\bm{0}$ is a $[M^' \times M^']$ matrix. Let ${\bm{\mu}}^d=(\mu_1^d,\mu_2^d,\cdots,\mu_{

Figures (21)

  • Figure 1: Two univariate expert distributions with estimates $\mu_1=4$ and $\mu_2=8$ on the unknown variable $\theta=8$. Expert 2 is more certain ($\sigma_2^2=1$) than expert 1 ($\sigma_1^2=3$).
  • Figure 2: Trade-off between generative coherence ($\uparrow$) and joint log-likelihoods ($\uparrow$) on the MNIST-SVHN-Text test set.
  • Figure 3: Generative quality gap for modality $m_0$ as a function of the number of modalities used to train the model.
  • Figure 4: Trade-off between generative coherence ($\uparrow$) and log-likelihood estimation ($\uparrow$), and between generative coherence and generative quality ($\downarrow$) for $\beta \in [1, 2.5, 5]$ on the PolyMNIST dataset.
  • Figure 5: The bars show the learned coefficients $\pi_k$ (left axis), while red lines show the average trace of the covariance matrix of the distribution $q({\bm{z}}|{\mathbb{X}}_k)$ (right axis). All values are averages over the subsets with the same cardinality.
  • ...and 16 more figures

Theorems & Definitions (9)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Lemma 2
  • Remark 1
  • Lemma 3
  • proof
  • proof
  • proof