Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hitchhiker's guide on the relation of Energy-Based Models with other generative models, sampling and statistical physics: a comprehensive review

Davide Carbone

TL;DR

Energy-Based Models (EBMs) provide a probabilistic framework where the target distribution is the Boltzmann-Gibbs density $\rho(x) \propto e^{-\beta U_\theta(x)}$ with partition function $Z_\theta$. This review argues for viewing EBMs as a natural interface between statistical physics, sampling theory, and modern generative models, contrasting them with VAEs, GANs, diffusion models, and normalizing flows. It surveys sampling-based and sampling-free training methods, including Langevin/MCMC approaches and score-based objectives, and explains how non-equilibrium ideas and free energy illuminate training dynamics and evaluation. The work highlights practical considerations, historical context, and open directions, advocating deeper physics-informed methodologies to advance interpretable and scalable generative modeling.

Abstract

Energy-Based Models have emerged as a powerful framework in the realm of generative modeling, offering a unique perspective that aligns closely with principles of statistical mechanics. This review aims to provide physicists with a comprehensive understanding of EBMs, delineating their connection to other generative models such as Generative Adversarial Networks, Variational Autoencoders, and Normalizing Flows. We explore the sampling techniques crucial for EBMs, including Markov Chain Monte Carlo (MCMC) methods, and draw parallels between EBM concepts and statistical mechanics, highlighting the significance of energy functions and partition functions. Furthermore, we delve into recent training methodologies for EBMs, covering recent advancements and their implications for enhanced model performance and efficiency. This review is designed to clarify the often complex interconnections between these models, which can be challenging due to the diverse communities working on the topic.

Hitchhiker's guide on the relation of Energy-Based Models with other generative models, sampling and statistical physics: a comprehensive review

TL;DR

Energy-Based Models (EBMs) provide a probabilistic framework where the target distribution is the Boltzmann-Gibbs density with partition function . This review argues for viewing EBMs as a natural interface between statistical physics, sampling theory, and modern generative models, contrasting them with VAEs, GANs, diffusion models, and normalizing flows. It surveys sampling-based and sampling-free training methods, including Langevin/MCMC approaches and score-based objectives, and explains how non-equilibrium ideas and free energy illuminate training dynamics and evaluation. The work highlights practical considerations, historical context, and open directions, advocating deeper physics-informed methodologies to advance interpretable and scalable generative modeling.

Abstract

Energy-Based Models have emerged as a powerful framework in the realm of generative modeling, offering a unique perspective that aligns closely with principles of statistical mechanics. This review aims to provide physicists with a comprehensive understanding of EBMs, delineating their connection to other generative models such as Generative Adversarial Networks, Variational Autoencoders, and Normalizing Flows. We explore the sampling techniques crucial for EBMs, including Markov Chain Monte Carlo (MCMC) methods, and draw parallels between EBM concepts and statistical mechanics, highlighting the significance of energy functions and partition functions. Furthermore, we delve into recent training methodologies for EBMs, covering recent advancements and their implications for enhanced model performance and efficiency. This review is designed to clarify the often complex interconnections between these models, which can be challenging due to the diverse communities working on the topic.
Paper Structure (23 sections, 19 theorems, 115 equations, 6 figures, 1 table, 2 algorithms)

This paper contains 23 sections, 19 theorems, 115 equations, 6 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Definition of EBM
  3. Cross-entropy minimization
  4. EBMs among generative models
  5. Variational Autoencoders
  6. Generative Adversarial Networks
  7. Diffusion Models
  8. Score-based diffusionsong2020score.
  9. Normalizing Flows
  10. Comparison with EBMs
  11. Why (and why not) EBMs in practice
  12. EBMs and sampling
  13. EBMs and physics
  14. Recent Developments in EBM training
  15. Sampling-based methods
  16. ...and 8 more sections

Key Result

Lemma 2.1

The following equality holds for any choice of PDFs $\rho_{1}$ and $\rho_2$

Figures (6)

  • Figure 1: Gaussian Mixture. Plot of PDF with sampled histogram and associated energy $U_\theta$.
  • Figure 2: Representation of an autoencoder, taken from https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73
  • Figure 3: Scheme of the structure of GANs, taken from https://sthalles.github.io/intro-to-gans/.
  • Figure 4: Comparison between KL divergence and Fisher divergence for the two bimodal gaussian mixtures in equation \ref{['eq:bim-gauss']}. The variable $z\in(0,\infty)$ is related to the relative mass of the two modes via a sigmoid function $\sigma(z)\in(0,1)$; the plots for $z<0$ are analogous by symmetry. Notice the different scales of the $y$ axes. The Monte Carlo estimation is performed using $N=100,1000,10000$ samples.
  • Figure 5: Schematic representation of forward and backward process in score-based diffusion. Image taken from song2020score.
  • ...and 1 more figures

Theorems & Definitions (49)

  • Remark 2.1
  • Definition 2.1: Convexity
  • Definition 2.2: Log-Concavity
  • Example 2.1
  • Definition 2.3
  • Lemma 2.1
  • Remark 2.2: Fundamental problem for EBM training
  • Definition 3.1: ELBO
  • Lemma 3.1
  • proof
  • ...and 39 more