Multi-Step Consistency Models: Fast Generation with Theoretical Guarantees

TL;DR

The paper provides a theoretical foundation for fast generation with consistency models by analyzing multi-step sampling along the probability-flow ODE, showing that a constant-step, noise-regularized scheme attains $O(\varepsilon^2)$ KL convergence in $O(\log(d/\varepsilon))$ steps under standard smoothness and score-estimation assumptions. It extends the results to non-smooth data distributions, obtaining a dimension-dependent but still favorable $O(d\log(d/\varepsilon))$ step count, and shows that accurate learning of the consistency function via distillation is achievable with fine discretization. The findings give explicit iteration- and step-size bounds, explain how added noise controls error accumulation, and position consistency-distillation as a practical training approach for fast, theoretically guaranteed generation. Overall, the work clarifies when and why consistency models can outperform SDE/ODE-based samplers in terms of inference efficiency, and provides guidance for designing training and inference protocols with provable guarantees.

Abstract

Consistency models have recently emerged as a compelling alternative to traditional SDE-based diffusion models. They offer a significant acceleration in generation by producing high-quality samples in very few steps. Despite their empirical success, a proper theoretic justification for their speed-up is still lacking. In this work, we address the gap by providing a theoretical analysis of consistency models capable of mapping inputs at a given time to arbitrary points along the reverse trajectory. We show that one can achieve a KL divergence of order $ O(\varepsilon^2) $ using only $ O\left(\log\left(\frac{d}{\varepsilon}\right)\right) $ iterations with a constant step size. Additionally, under minimal assumptions on the data distribution (non smooth case) an increasingly common setting in recent diffusion model analyses we show that a similar KL convergence guarantee can be obtained, with the number of steps scaling as $ O\left(d \log\left(\frac{d}{\varepsilon}\right)\right) $. Going further, we also provide a theoretical analysis for estimation of such consistency models, concluding that accurate learning is feasible using small discretization steps, both in smooth and non-smooth settings. Notably, our results for the non-smooth case yield best in class convergence rates compared to existing SDE or ODE based analyses under minimal assumptions.

Figures (1)

• Figure 1: Demonstrating the step 3 ($\hat{f}$) and 5 ($\mathcal{N}$) of our algorithm w.r.t. the reverse ODE.

Theorems & Definitions (31)

• Theorem 3.1
• Corollary 3.2
• Theorem 3.3
• Remark 1
• Corollary 3.4
• Theorem 3.5
• proof
• Lemma 3.6
• Theorem 3.7
• Lemma B.1