Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Multimodal Latent Space with EBM Prior and MCMC Inference

Shiyu Yuan, Carlo Lipizzi, Tian Han

TL;DR

This work tackles the limitations of uni-modal priors in multimodal generation by introducing an expressive latent-space energy-based model prior $p_\alpha(z)$ and leveraging short-run Langevin MCMC to better approximate the posterior. By integrating a mixture-of-experts style MOE aggregation with an EBM prior, the method trains via maximum likelihood with an energy term, enabling improved learning of shared latent representations across modalities. Empirical results on PolyMNIST demonstrate improvements in joint and cross-modal coherence and competitive perceptual quality, validating the benefit of combining an expressive latent prior with MCMC-based posterior sampling. Overall, the approach offers a principled framework for capturing multimodal complexity in latent spaces, with practical gains in cross-modal generation tasks.

Abstract

Multimodal generative models are crucial for various applications. We propose an approach that combines an expressive energy-based model (EBM) prior with Markov Chain Monte Carlo (MCMC) inference in the latent space for multimodal generation. The EBM prior acts as an informative guide, while MCMC inference, specifically through short-run Langevin dynamics, brings the posterior distribution closer to its true form. This method not only provides an expressive prior to better capture the complexity of multimodality but also improves the learning of shared latent variables for more coherent generation across modalities. Our proposed method is supported by empirical experiments, underscoring the effectiveness of our EBM prior with MCMC inference in enhancing cross-modal and joint generative tasks in multimodal contexts.

Learning Multimodal Latent Space with EBM Prior and MCMC Inference

TL;DR

This work tackles the limitations of uni-modal priors in multimodal generation by introducing an expressive latent-space energy-based model prior and leveraging short-run Langevin MCMC to better approximate the posterior. By integrating a mixture-of-experts style MOE aggregation with an EBM prior, the method trains via maximum likelihood with an energy term, enabling improved learning of shared latent representations across modalities. Empirical results on PolyMNIST demonstrate improvements in joint and cross-modal coherence and competitive perceptual quality, validating the benefit of combining an expressive latent prior with MCMC-based posterior sampling. Overall, the approach offers a principled framework for capturing multimodal complexity in latent spaces, with practical gains in cross-modal generation tasks.

Abstract

Multimodal generative models are crucial for various applications. We propose an approach that combines an expressive energy-based model (EBM) prior with Markov Chain Monte Carlo (MCMC) inference in the latent space for multimodal generation. The EBM prior acts as an informative guide, while MCMC inference, specifically through short-run Langevin dynamics, brings the posterior distribution closer to its true form. This method not only provides an expressive prior to better capture the complexity of multimodality but also improves the learning of shared latent variables for more coherent generation across modalities. Our proposed method is supported by empirical experiments, underscoring the effectiveness of our EBM prior with MCMC inference in enhancing cross-modal and joint generative tasks in multimodal contexts.
Paper Structure (14 sections, 10 equations, 3 figures, 3 tables)

This paper contains 14 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Generative Models
  4. Expressive Prior
  5. MCMC-based Inference
  6. Methodology
  7. Preliminaries
  8. Method
  9. Experiment
  10. Dataset and Experiment Settings
  11. EBM Prior with MCMC Inference
  12. Generation Comparison between Variational Inference and MCMC Inference
  13. Ablation Studies
  14. Future work

Figures (3)

  • Figure 1: Joint Generation
  • Figure 2: Cross Generation: from right to left are EBM on MMVAE(MOE), MMVAE$+$, EBM on MMVAE$+$, MOE-EBM on MMVAE$+$
  • Figure 3: (a) Comparative visualization of generation quality before and after LD refinement. (b) Generation improvement during Markov transitions using LD.