Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Forgetting and Stability of Score-based Generative models

Stanislas Strasman, Gabriel Cardoso, Sylvain Le Corff, Vincent Lemaire, Antonio Ocello

TL;DR

This work addresses the stability and long-time behavior of score-based diffusion models by proving quantitative bounds on sampling error through a Harris stability framework. It shows that the backward diffusion process satisfies a Lyapunov drift condition and a localized minorization, which together induce contraction of the Markov semigroup in a weighted total variation metric, thereby enabling forgetting of initialization and discretization errors along the sampling trajectory. The authors derive explicit constants and relax global Lipschitz assumptions, allowing nonconvex and multimodal data by leveraging a dissipativity condition on the score. They also provide Gaussian-case contractions with closed-form expressions and validate the theory with numerical experiments demonstrating robustness to initialization and local score-approximation errors. Overall, the paper furnishes a principled probabilistic framework for analyzing error propagation in score-based generative models, guiding hyperparameter choices and stability considerations in practice.

Abstract

Understanding the stability and long-time behavior of generative models is a fundamental problem in modern machine learning. This paper provides quantitative bounds on the sampling error of score-based generative models by leveraging stability and forgetting properties of the Markov chain associated with the reverse-time dynamics. Under weak assumptions, we provide the two structural properties to ensure the propagation of initialization and discretization errors of the backward process: a Lyapunov drift condition and a Doeblin-type minorization condition. A practical consequence is quantitative stability of the sampling procedure, as the reverse diffusion dynamics induces a contraction mechanism along the sampling trajectory. Our results clarify the role of stochastic dynamics in score-based models and provide a principled framework for analyzing propagation of errors in such approaches.

On Forgetting and Stability of Score-based Generative models

TL;DR

This work addresses the stability and long-time behavior of score-based diffusion models by proving quantitative bounds on sampling error through a Harris stability framework. It shows that the backward diffusion process satisfies a Lyapunov drift condition and a localized minorization, which together induce contraction of the Markov semigroup in a weighted total variation metric, thereby enabling forgetting of initialization and discretization errors along the sampling trajectory. The authors derive explicit constants and relax global Lipschitz assumptions, allowing nonconvex and multimodal data by leveraging a dissipativity condition on the score. They also provide Gaussian-case contractions with closed-form expressions and validate the theory with numerical experiments demonstrating robustness to initialization and local score-approximation errors. Overall, the paper furnishes a principled probabilistic framework for analyzing error propagation in score-based generative models, guiding hyperparameter choices and stability considerations in practice.

Abstract

Understanding the stability and long-time behavior of generative models is a fundamental problem in modern machine learning. This paper provides quantitative bounds on the sampling error of score-based generative models by leveraging stability and forgetting properties of the Markov chain associated with the reverse-time dynamics. Under weak assumptions, we provide the two structural properties to ensure the propagation of initialization and discretization errors of the backward process: a Lyapunov drift condition and a Doeblin-type minorization condition. A practical consequence is quantitative stability of the sampling procedure, as the reverse diffusion dynamics induces a contraction mechanism along the sampling trajectory. Our results clarify the role of stochastic dynamics in score-based models and provide a principled framework for analyzing propagation of errors in such approaches.
Paper Structure (65 sections, 32 theorems, 288 equations, 6 figures, 2 algorithms)

This paper contains 65 sections, 32 theorems, 288 equations, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Background
  4. Notations
  5. Score-based generative models.
  6. Backward process approximations.
  7. Generation error analysis.
  8. Forgetting of the backward process
  9. Harris-type stability and weighted total variation distance.
  10. Assumptions on the data distribution.
  11. Comparison with recent literature.
  12. Lyapunov drift for the backward chain.
  13. Minorization for the backward chain.
  14. Forgetting via Harris contraction.
  15. Explicit Gaussian contraction.
  16. ...and 50 more sections

Key Result

Proposition 3.2

[proposition]prop:backward_drift_lyapunov Suppose that assump:p0 holds and let $\operatorname{V}_{\ell} \!\left(\mathbf{x}\right) := \left\|\mathbf{x}\right\|^{\ell}$ for $\ell\ge2$. Then, there exist continuous functions $\tilde{\gamma}_{\cdot,\ell} , \tilde{\kappa}_{\cdot,\ell} :[0,T]\to\mathbb{ where $\lambda_{t|s}^{}:= \exp \left(-\int_s^t \tilde{\gamma}_{v,\ell} \,\mathrm{d} v\right)$ and

Figures (6)

  • Figure 1: Initialization bias in the GMM case for several noise levels $\sigma_t$ and with $\lambda=20$. Red points are samples from $\pi_{\mathrm{data}}$ and blue points the output of the biased initialization experiment.
  • Figure 2: Local perturbation of the score in the GMM case for several noise levels $\sigma_t$ and with $\lambda=50$. Red points are samples from $\pi_{\mathrm{data}}$ and blue points the output of the perturbed score experiment.
  • Figure 3: Max SW as a function of the noise level and $\lambda$ in the case of initialization perturbation. We use the forward-time convention ($t=0$ is the data distribution).
  • Figure 4: Max SW as a function of the noise level and perturbation magnitude $\lambda$ for the score-perturbation experiment. We use the forward-time convention ($t=0$ is the data distribution).
  • Figure 5: Sensitivity to initialization for isotropic, heteroscedastic, and correlated Gaussian targets. Curves report the mean over 5 independent replications of Algorithm \ref{['alg:init-bias']} ($T=1$, $M=30{,}000$, $h=2.5\times10^{-3}$); shaded regions indicate $\pm 1$ standard deviation across replications. The horizontal axis is the forward diffusion time $t$. Each dotted point on the lines correspond to a perturbation time and the bias magnitude corresponds to the choice of $\lambda$.
  • ...and 1 more figures

Theorems & Definitions (70)

  • Proposition 3.2
  • proof
  • Proposition 3.3
  • proof
  • Proposition 3.4
  • proof
  • Remark 4.2
  • Theorem 4.3
  • proof
  • Lemma 1.1
  • ...and 60 more