Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Modelling with High-Order Langevin Dynamics

Ziqiang Shi, Rujie Liu

TL;DR

This work introduces HOLD, a fast, high-order diffusion framework that augments the data state with velocity and acceleration via a third-order Langevin forward process. By combining two Hamiltonians with an OU component and employing Block Coordinate Score Matching and Lie-Trotter splitting, HOLD achieves dramatically faster mixing and high-quality sample generation, demonstrated on CIFAR-10 and CelebA-HQ-256 with state-of-the-art or competitive metrics. The approach provides a flexible objective, robust ablations, and a principled likelihood bound, expanding the applicability of score-based diffusion models to richer latent dynamics and multiple data modalities.

Abstract

Diffusion generative modelling (DGM) based on stochastic differential equations (SDEs) with score matching has achieved unprecedented results in data generation. In this paper, we propose a novel fast high-quality generative modelling method based on high-order Langevin dynamics (HOLD) with score matching. This motive is proved by third-order Langevin dynamics. By augmenting the previous SDEs, e.g. variance exploding or variance preserving SDEs for single-data variable processes, HOLD can simultaneously model position, velocity, and acceleration, thereby improving the quality and speed of the data generation at the same time. HOLD is composed of one Ornstein-Uhlenbeck process and two Hamiltonians, which reduce the mixing time by two orders of magnitude. Empirical experiments for unconditional image generation on the public data set CIFAR-10 and CelebA-HQ show that the effect is significant in both Frechet inception distance (FID) and negative log-likelihood, and achieves the state-of-the-art FID of 1.85 on CIFAR-10.

Generative Modelling with High-Order Langevin Dynamics

TL;DR

This work introduces HOLD, a fast, high-order diffusion framework that augments the data state with velocity and acceleration via a third-order Langevin forward process. By combining two Hamiltonians with an OU component and employing Block Coordinate Score Matching and Lie-Trotter splitting, HOLD achieves dramatically faster mixing and high-quality sample generation, demonstrated on CIFAR-10 and CelebA-HQ-256 with state-of-the-art or competitive metrics. The approach provides a flexible objective, robust ablations, and a principled likelihood bound, expanding the applicability of score-based diffusion models to richer latent dynamics and multiple data modalities.

Abstract

Diffusion generative modelling (DGM) based on stochastic differential equations (SDEs) with score matching has achieved unprecedented results in data generation. In this paper, we propose a novel fast high-quality generative modelling method based on high-order Langevin dynamics (HOLD) with score matching. This motive is proved by third-order Langevin dynamics. By augmenting the previous SDEs, e.g. variance exploding or variance preserving SDEs for single-data variable processes, HOLD can simultaneously model position, velocity, and acceleration, thereby improving the quality and speed of the data generation at the same time. HOLD is composed of one Ornstein-Uhlenbeck process and two Hamiltonians, which reduce the mixing time by two orders of magnitude. Empirical experiments for unconditional image generation on the public data set CIFAR-10 and CelebA-HQ show that the effect is significant in both Frechet inception distance (FID) and negative log-likelihood, and achieves the state-of-the-art FID of 1.85 on CIFAR-10.
Paper Structure (28 sections, 1 theorem, 99 equations, 11 figures, 3 tables)

This paper contains 28 sections, 1 theorem, 99 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Generative Modelling with High-Order Langevin Dynamics
  4. Third-Order Langevin Dynamics
  5. Block Coordinate Score Matching
  6. Lie-Trotter Sampler (for HOLD-DGM)
  7. Image Restoration with HOLD
  8. Experiments
  9. 2D multi-swiss roll generation
  10. Image Generation
  11. Ablation Studies
  12. Conclusion
  13. A Lemma
  14. High-Order Langevin Dynamics
  15. Perturbation Kernel
  16. ...and 13 more sections

Key Result

Lemma 1

(Exponential of Kronecker product) Let ${\bm{A}}\in\mathbb{R}^{d \times d}$ and ${\bm{I}}$ is identity matrix, then $\exp({\bm{A}}\otimes {\bm{I}})=\exp({\bm{A}})\otimes {\bm{I}}$.

Figures (11)

  • Figure 1: Comparison of solution paths between HOLD-based DGM and other methods on 1D data. The paths in white are obtained by integrating the corresponding ODE flows. The coloured paths are obtained by solving the corresponding backward generative SDE with Euler-Maruyama method.
  • Figure 2: In 1D case, the solution (data generation) paths of the position (white), velocity (yellow), and acceleration (green) variables in HOLD.
  • Figure 3: Comparison of different moments (number of steps) on the solution path generated by CLD and HOLD-based DGM on 2D multi-Swiss rolls.
  • Figure 4: Comparison of 2D multi-Swiss rolls generated by three DGMs based on VP SDE, CLD and HOLD.
  • Figure 5: Generated CIFAR-10 samples without cherry-picking.
  • ...and 6 more figures

Theorems & Definitions (2)

  • Lemma 1
  • proof