Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Of Consistency Model With Multistep Sampling Under General Data Assumptions

Yiding Chen, Yiyi Zhang, Owen Oertell, Wen Sun

TL;DR

The paper analyzes convergence of consistency models (CMs) with multistep sampling under mild data assumptions across a broad class of forward diffusion processes. It proves that CM-generated samples approximate the data distribution in $W_2$ when the data has bounded support or light tails, and, with an added smoothing step, in TV under data-smoothness conditions. Theoretical guarantees are paired with two case studies (Ornstein–Uhlenbeck and variance-exploding SDEs) to illustrate sampling schedules and the trade-offs of multistep inference. By leveraging approximate self-consistency and the probability-flow ODE framework, the work relaxes prior Lipschitz requirements and provides a dimension-free, general framework for CM convergence in diffusion-based generative modeling.

Abstract

Diffusion models accomplish remarkable success in data generation tasks across various domains. However, the iterative sampling process is computationally expensive. Consistency models are proposed to learn consistency functions to map from noise to data directly, which allows one-step fast data generation and multistep sampling to improve sample quality. In this paper, we study the convergence of consistency models when the self-consistency property holds approximately under the training distribution. Our analysis requires only mild data assumption and applies to a family of forward processes. When the target data distribution has bounded support or has tails that decay sufficiently fast, we show that the samples generated by the consistency model are close to the target distribution in Wasserstein distance; when the target distribution satisfies some smoothness assumption, we show that with an additional perturbation step for smoothing, the generated samples are close to the target distribution in total variation distance. We provide two case studies with commonly chosen forward processes to demonstrate the benefit of multistep sampling.

Convergence Of Consistency Model With Multistep Sampling Under General Data Assumptions

TL;DR

The paper analyzes convergence of consistency models (CMs) with multistep sampling under mild data assumptions across a broad class of forward diffusion processes. It proves that CM-generated samples approximate the data distribution in when the data has bounded support or light tails, and, with an added smoothing step, in TV under data-smoothness conditions. Theoretical guarantees are paired with two case studies (Ornstein–Uhlenbeck and variance-exploding SDEs) to illustrate sampling schedules and the trade-offs of multistep inference. By leveraging approximate self-consistency and the probability-flow ODE framework, the work relaxes prior Lipschitz requirements and provides a dimension-free, general framework for CM convergence in diffusion-based generative modeling.

Abstract

Diffusion models accomplish remarkable success in data generation tasks across various domains. However, the iterative sampling process is computationally expensive. Consistency models are proposed to learn consistency functions to map from noise to data directly, which allows one-step fast data generation and multistep sampling to improve sample quality. In this paper, we study the convergence of consistency models when the self-consistency property holds approximately under the training distribution. Our analysis requires only mild data assumption and applies to a family of forward processes. When the target data distribution has bounded support or has tails that decay sufficiently fast, we show that the samples generated by the consistency model are close to the target distribution in Wasserstein distance; when the target distribution satisfies some smoothness assumption, we show that with an additional perturbation step for smoothing, the generated samples are close to the target distribution in total variation distance. We provide two case studies with commonly chosen forward processes to demonstrate the benefit of multistep sampling.
Paper Structure (35 sections, 11 theorems, 67 equations, 2 figures, 1 algorithm)

This paper contains 35 sections, 11 theorems, 67 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Mismatch on the initial starting distributions:
  3. Approximate self-consistency:
  4. Complexity of multistep sampling:
  5. Our contributions
  6. Guarantees in Wasserstein distance.
  7. Guarantees in Total Variation distance.
  8. Related work
  9. Preliminaries
  10. Score-based generative model:
  11. Consistency models:
  12. Multistep sampling:
  13. Performance metric:
  14. Main results
  15. Guarantees in Wasserstein metric
  16. ...and 20 more sections

Key Result

Theorem 1

Suppose the consistency function estimate is $\epsilon^2$ accurate and the support of the target distribution ${P_{\textrm{data}}}$ is bounded by $R$, then one-step sampling returns a distribution that is $(*){\epsilon\log\frac{R^3}{\epsilon^2}}$-close to ${P_{\textrm{data}}}$ in $W_2$ distance; two

Figures (2)

  • Figure 1: Smoothing by additional perturbation
  • Figure 2: $W_2$ error in multi-step sampling.

Theorems & Definitions (11)

  • Theorem 1: informal, see Theorem \ref{['thm:w2-general-bounded']} and Corollary \ref{['coro:W2-2step-ou']}
  • Theorem 2: $W_2$ error for distributions with bounded support
  • Theorem 3: TV error for distributions under smoothness assumption
  • Corollary 1: Two-step sampling with OU process
  • Corollary 2: Multistep sampling with the variance exploding SDE
  • Lemma 1: Decomposition of KL
  • Lemma 2
  • Theorem 4: $W_2$ error for distributions with tail condition
  • Lemma 3
  • Lemma 4: Gaussian perturbation on a smooth distribution, a variant of Lemma 6.4 of lee2023convergence
  • ...and 1 more