Contractive Diffusion Probabilistic Models
Wenpin Tang, Hanyang Zhao
TL;DR
This work introduces Contractive Diffusion Probabilistic Models (CDPMs), a framework that enforces contractive backward sampling to curb the propagation of score-matching and discretization errors in diffusion models. It develops theoretical Wasserstein-2 bounds, shows how contraction can be achieved via design choices (contractive OU, VP, and subVP SDEs), and connects CDPMs to VE through a transformation that leverages pretrained scores without retraining. Empirically, CDPMs demonstrate robustness and improved performance across 1D, Swiss Roll, MNIST, CIFAR-10, and AFHQ data, notably achieving competitive CIFAR-10 results while requiring no retraining. The results advocate for incorporating contraction into DPM design as a principled path to more reliable and efficient generative modeling.
Abstract
Diffusion probabilistic models (DPMs) have emerged as a promising technique in generative modeling. The success of DPMs relies on two ingredients: time reversal of diffusion processes and score matching. In view of possibly unguaranteed score matching, we propose a new criterion -- the contraction property of backward sampling in the design of DPMs, leading to a novel class of contractive DPMs (CDPMs). Our key insight is that, the contraction property can provably narrow score matching errors and discretization errors, thus our proposed CDPMs are robust to both sources of error. For practical use, we show that CDPM can leverage weights of pretrained DPMs by a simple transformation, and does not need retraining. We corroborated our approach by experiments on synthetic 1-dim examples, Swiss Roll, MNIST, CIFAR-10 32$\times$32 and AFHQ 64$\times$64 dataset. Notably, CDPM steadily improves the performance of baseline score-based diffusion models.