Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders

Fotios Lygerakis, Elmar Rueckert

TL;DR

This work targets limitations of the standard ELBO in VAEs when priors are nonanalytic or unknown. It introduces ED-VAE, a reformulation that explicitly decomposes the ELBO into reconstruction, mutual information, and a marginal KL term split into entropy and cross-entropy $H[q(z)]$ and $H[q(z), p(z)]$, enabling flexible priors. Mutual information is bounded via InfoNCE to encourage informative latent codes, and the total loss combines reconstruction, InfoNCE, entropy, and cross-entropy terms. Experiments on synthetic datasets with Gaussian and complex non-Gaussian priors demonstrate that ED-VAE achieves higher ELBO and better prior alignment, improving reconstruction quality and latent space interpretability, highlighting its potential for adaptable generative modeling with domain-specific priors.

Abstract

Traditional Variational Autoencoders (VAEs) are constrained by the limitations of the Evidence Lower Bound (ELBO) formulation, particularly when utilizing simplistic, non-analytic, or unknown prior distributions. These limitations inhibit the VAE's ability to generate high-quality samples and provide clear, interpretable latent representations. This work introduces the Entropy Decomposed Variational Autoencoder (ED-VAE), a novel re-formulation of the ELBO that explicitly includes entropy and cross-entropy components. This reformulation significantly enhances model flexibility, allowing for the integration of complex and non-standard priors. By providing more detailed control over the encoding and regularization of latent spaces, ED-VAE not only improves interpretability but also effectively captures the complex interactions between latent variables and observed data, thus leading to better generative performance.

ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders

TL;DR

This work targets limitations of the standard ELBO in VAEs when priors are nonanalytic or unknown. It introduces ED-VAE, a reformulation that explicitly decomposes the ELBO into reconstruction, mutual information, and a marginal KL term split into entropy and cross-entropy and , enabling flexible priors. Mutual information is bounded via InfoNCE to encourage informative latent codes, and the total loss combines reconstruction, InfoNCE, entropy, and cross-entropy terms. Experiments on synthetic datasets with Gaussian and complex non-Gaussian priors demonstrate that ED-VAE achieves higher ELBO and better prior alignment, improving reconstruction quality and latent space interpretability, highlighting its potential for adaptable generative modeling with domain-specific priors.

Abstract

Traditional Variational Autoencoders (VAEs) are constrained by the limitations of the Evidence Lower Bound (ELBO) formulation, particularly when utilizing simplistic, non-analytic, or unknown prior distributions. These limitations inhibit the VAE's ability to generate high-quality samples and provide clear, interpretable latent representations. This work introduces the Entropy Decomposed Variational Autoencoder (ED-VAE), a novel re-formulation of the ELBO that explicitly includes entropy and cross-entropy components. This reformulation significantly enhances model flexibility, allowing for the integration of complex and non-standard priors. By providing more detailed control over the encoding and regularization of latent spaces, ED-VAE not only improves interpretability but also effectively captures the complex interactions between latent variables and observed data, thus leading to better generative performance.
Paper Structure (20 sections, 13 equations, 1 figure, 2 tables)

This paper contains 20 sections, 13 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Original ELBO Formulation
  5. Entropy Decomposed ELBO
  6. ELBO Optimization
  7. Experiments
  8. Synthetic datasets
  9. Dataset 1: Gaussian Prior
  10. Dataset 2: Complex Non-Gaussian Prior
  11. Model Configurations
  12. Experimental Setup
  13. Results
  14. Discussion
  15. Supplementary Material
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Visual description of the Evidence Lower BOund (ELBO) in a ED-VAE. Figure \ref{['fig:ed_vae_graph']} describes the minimization of ELBO, which we show consists of a mutual information, an entropy and a cross-entropy term, as shown in Section \ref{['sec:entropy_dec']}.