Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Abstractions in Generative Diffusion Models

Giulio Franzese, Mattia Martini, Giulio Corallo, Paolo Papotti, Pietro Michiardi

TL;DR

It is shown that diffusion models can be interpreted as a system of SDE, describing a non-linear filter where unobservable latent abstractions steer the dynamics of an observable measurement process.

Abstract

In this work we study how diffusion-based generative models produce high-dimensional data, such as an image, by implicitly relying on a manifestation of a low-dimensional set of latent abstractions, that guide the generative process. We present a novel theoretical framework that extends NLF, and that offers a unique perspective on SDE-based generative models. The development of our theory relies on a novel formulation of the joint (state and measurement) dynamics, and an information-theoretic measure of the influence of the system state on the measurement process. According to our theory, diffusion models can be cast as a system of SDE, describing a non-linear filter in which the evolution of unobservable latent abstractions steers the dynamics of an observable measurement process (corresponding to the generative pathways). In addition, we present an empirical study to validate our theory and previous empirical results on the emergence of latent abstractions at different stages of the generative process.

Latent Abstractions in Generative Diffusion Models

TL;DR

It is shown that diffusion models can be interpreted as a system of SDE, describing a non-linear filter where unobservable latent abstractions steer the dynamics of an observable measurement process.

Abstract

In this work we study how diffusion-based generative models produce high-dimensional data, such as an image, by implicitly relying on a manifestation of a low-dimensional set of latent abstractions, that guide the generative process. We present a novel theoretical framework that extends NLF, and that offers a unique perspective on SDE-based generative models. The development of our theory relies on a novel formulation of the joint (state and measurement) dynamics, and an information-theoretic measure of the influence of the system state on the measurement process. According to our theory, diffusion models can be cast as a system of SDE, describing a non-linear filter in which the evolution of unobservable latent abstractions steers the dynamics of an observable measurement process (corresponding to the generative pathways). In addition, we present an empirical study to validate our theory and previous empirical results on the emergence of latent abstractions at different stages of the generative process.
Paper Structure (29 sections, 9 theorems, 60 equations, 5 figures, 5 tables)

This paper contains 29 sections, 9 theorems, 60 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Nonlinear Filtering
  3. Technical preliminaries
  4. A-Posteriori Measure and Mutual Information
  5. Generative Modelling
  6. An informal summary of the results
  7. Empirical Evidence
  8. Conclusion
  9. Acknowledgments
  10. Assumptions
  11. Proof of \ref{['innovation_theo']}
  12. Proof of \ref{['theo_girsa']}
  13. Proof of \ref{['kushner_eq']}
  14. Proof of \ref{['theo_mi']}
  15. Part 1
  16. ...and 14 more sections

Key Result

Theorem 1

[Thm 2.1 bain2009fundamentals] Consider the probability triplet $(\Omega,\mathcal{F},\mathrm{P})$, the metric space $\mathcal{S}$ and its Borel sigma-algebra $\mathcal{B}(\mathcal{S})$. There exists a (probability measure valued $\mathcal{P}(\mathcal{S})$) process $\{ { {\pi} } _{0\leq t\leq T},\mat

Figures (5)

  • Figure 1: Graphical intuition for our results: nonlinear filtering (left) and generative modelling (right).
  • Figure 2: Effect of the variance-exploding schedule on an image corrupted by noise with intensity $\tau$.
  • Figure 3: Mutual information, Entropy across forked generative pathways, and Probing results as functions of $\tau$.
  • Figure 4: Visualization of the forking experiment with $\texttt{num\_forks} = 4$ and one initial seed. The image at time $\tau = 0.4$ is quite noisy. In the final generations after forking, the images exhibit coherence in the labels shape, wall hue, floor hue, and object hue. However, there is variation in orientation and scale.
  • Figure 5: Log-probability accuracy of linear classifiers at $\tau$. 'Feature map' classifiers are trained on network features; 'Noisy Image' trained on noisy images; 'Random Guess' is the baseline for random guessing.

Theorems & Definitions (9)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Theorem 8
  • Lemma 1