Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memorization and Regularization in Generative Diffusion Models

Ricardo Baptista, Agnimitra Dasgupta, Nikola B. Kovachki, Assad Oberai, Andrew M. Stuart

TL;DR

This work analyzes memorization in score-based diffusion models that learn the score from empirical data, showing that the empirical score minimizer corresponds to the score of a time-dependent Gaussian mixture and can cause the model to reproduce training samples. It introduces an inverse-problem perspective on score learning, proving that under both variance-exploding and variance-preserving forward processes, reverse trajectories with the empirical score converge to training points or to Voronoi boundaries, a phenomenon termed memorization. The authors then develop principled regularization approaches—Tikhonov regularization with a time-scale dependent penalty, empirical Bayes regularization, and neural-network-based strategies including early stopping and under-parameterization—demonstrating that these regularizers mitigate memorization in low-dimensional and image data, and extending the framework to conditional sampling. The results provide a mathematical lens on why practical diffusion models often generalize despite empirical optimization and outline concrete pathways to enforce sampling diversity and privacy through regularization. Overall, the paper lays a theory-grounded foundation for principled regularization of score-based generative models and offers practical guidance for managing memorization in both unconditional and conditional settings.

Abstract

Diffusion models have emerged as a powerful framework for generative modeling. At the heart of the methodology is score matching: learning gradients of families of log-densities for noisy versions of the data distribution at different scales. When the loss function adopted in score matching is evaluated using empirical data, rather than the population loss, the minimizer corresponds to the score of a time-dependent Gaussian mixture. However, use of this analytically tractable minimizer leads to data memorization: in both unconditioned and conditioned settings, the generative model returns the training samples. This paper contains an analysis of the dynamical mechanism underlying memorization. The analysis highlights the need for regularization to avoid reproducing the analytically tractable minimizer; and, in so doing, lays the foundations for a principled understanding of how to regularize. Numerical experiments investigate the properties of: (i) Tikhonov regularization; (ii) regularization designed to promote asymptotic consistency; and (iii) regularizations induced by under-parameterization of a neural network or by early stopping when training a neural network. These experiments are evaluated in the context of memorization, and directions for future development of regularization are highlighted.

Memorization and Regularization in Generative Diffusion Models

TL;DR

This work analyzes memorization in score-based diffusion models that learn the score from empirical data, showing that the empirical score minimizer corresponds to the score of a time-dependent Gaussian mixture and can cause the model to reproduce training samples. It introduces an inverse-problem perspective on score learning, proving that under both variance-exploding and variance-preserving forward processes, reverse trajectories with the empirical score converge to training points or to Voronoi boundaries, a phenomenon termed memorization. The authors then develop principled regularization approaches—Tikhonov regularization with a time-scale dependent penalty, empirical Bayes regularization, and neural-network-based strategies including early stopping and under-parameterization—demonstrating that these regularizers mitigate memorization in low-dimensional and image data, and extending the framework to conditional sampling. The results provide a mathematical lens on why practical diffusion models often generalize despite empirical optimization and outline concrete pathways to enforce sampling diversity and privacy through regularization. Overall, the paper lays a theory-grounded foundation for principled regularization of score-based generative models and offers practical guidance for managing memorization in both unconditional and conditional settings.

Abstract

Diffusion models have emerged as a powerful framework for generative modeling. At the heart of the methodology is score matching: learning gradients of families of log-densities for noisy versions of the data distribution at different scales. When the loss function adopted in score matching is evaluated using empirical data, rather than the population loss, the minimizer corresponds to the score of a time-dependent Gaussian mixture. However, use of this analytically tractable minimizer leads to data memorization: in both unconditioned and conditioned settings, the generative model returns the training samples. This paper contains an analysis of the dynamical mechanism underlying memorization. The analysis highlights the need for regularization to avoid reproducing the analytically tractable minimizer; and, in so doing, lays the foundations for a principled understanding of how to regularize. Numerical experiments investigate the properties of: (i) Tikhonov regularization; (ii) regularization designed to promote asymptotic consistency; and (iii) regularizations induced by under-parameterization of a neural network or by early stopping when training a neural network. These experiments are evaluated in the context of memorization, and directions for future development of regularization are highlighted.
Paper Structure (51 sections, 23 theorems, 192 equations, 20 figures)

This paper contains 51 sections, 23 theorems, 192 equations, 20 figures.

Table of Contents

  1. Introduction
  2. Context
  3. Contributions and Overview
  4. Literature Review
  5. Score-Based Generative Modeling
  6. Forward Process
  7. Reverse Process
  8. The Reverse Process In Practice
  9. Approximation of the Score
  10. Score Matching
  11. Properties of the Empirical Score
  12. Score Matching In Practice
  13. Learning The Noise
  14. Theory of Memorization
  15. Set-Up and Main Theorem
  16. ...and 36 more sections

Key Result

Lemma 3.1

For non-negative $\beta \colon [0,T] \rightarrow \mathbb{R}$ and strictly positive $g \colon [0,T] \rightarrow \mathbb{R}_{\geq 0}$, the solution $x(t) \in \mathbb{R}^d$ for $t \geq 0$ of the SDE has marginal law at each time $t$ given by

Figures (20)

  • Figure 1: Convergence of the reverse ODE trajectories for the variance exploding process to the data points in red when using the empirical score function starting from four initial conditions. The Voronoi tessellation is plotted for $N = 20$ samples in blue.
  • Figure 2: Convergence rate of the reverse ODE solutions to the data points for the variance exploding process with the empirical score function for 30 independent trajectories in the transformed time $s$ (left) and original time $t$ (right).
  • Figure 3: Trajectories of the reverse ODE for the variance exploding process with $N=2$ (left) and $N=4$ (right) data points. The trajectories start from a square of width $2\sigma(T)$. In both settings, we observe that most trajectories converge to the data points in blue, while some remain on the hyperplanes between the data for all time.
  • Figure 4: $100$ generated samples $x(0)$ found by integrating the reverse ODE \ref{['eq:mf_ode_empR']} backwards from $T=1.$ Blue dots comprise the data set; red circles are the generated points. The Tikhonov regularized score function is used with values $c = 0.0001$ ( left), $c = 0.01$ ( middle) and $c = 0.1$ ( right). The smallest value of $c$ exhibits collapse onto the data set; the larger values of $c$ prevent memorization.
  • Figure 5: The effect of the regularization parameter on the memorization phenomenon with Tikhonov regularization. The setting is identical to that in Figure \ref{['fig:tikonov_regularized']}.
  • ...and 15 more figures

Theorems & Definitions (62)

  • Example 2.1
  • Example 2.2
  • Example 2.3
  • Example 2.4
  • Example 2.5
  • Example 2.6
  • Lemma 3.1
  • Theorem 3.2
  • Remark 3.3
  • Example 3.4
  • ...and 52 more