Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DDPM Score Matching and Distribution Learning

Sinho Chewi, Alkis Kalavasis, Anay Mehrotra, Omar Montasser

TL;DR

This framework offers the first principled method to prove computational lower bounds for score estimation across general distributions, and establishes cryptographic lower bounds for score estimation in general Gaussian mixture models, conceptually recovering Song's result and advancing his key open problem.

Abstract

Score estimation is the backbone of score-based generative models (SGMs), especially denoising diffusion probabilistic models (DDPMs). A key result in this area shows that with accurate score estimates, SGMs can efficiently generate samples from any realistic data distribution (Chen et al., ICLR'23; Lee et al., ALT'23). This distribution learning result, where the learned distribution is implicitly that of the sampler's output, does not explain how score estimation relates to classical tasks of parameter and density estimation. This paper introduces a framework that reduces score estimation to these two tasks, with various implications for statistical and computational learning theory: Parameter Estimation: Koehler et al. (ICLR'23) demonstrate that a score-matching variant is statistically inefficient for the parametric estimation of multimodal densities common in practice. In contrast, we show that under mild conditions, denoising score-matching in DDPMs is asymptotically efficient. Density Estimation: By linking generation to score estimation, we lift existing score estimation guarantees to $(ε,δ)$-PAC density estimation, i.e., a function approximating the target log-density within $ε$ on all but a $δ$-fraction of the space. We provide (i) minimax rates for density estimation over Hölder classes and (ii) a quasi-polynomial PAC density estimation algorithm for the classical Gaussian location mixture model, building on and addressing an open problem from Gatmiry et al. (arXiv'24). Lower Bounds for Score Estimation: Our framework offers the first principled method to prove computational lower bounds for score estimation across general distributions. As an application, we establish cryptographic lower bounds for score estimation in general Gaussian mixture models, conceptually recovering Song's (NeurIPS'24) result and advancing his key open problem.

DDPM Score Matching and Distribution Learning

TL;DR

This framework offers the first principled method to prove computational lower bounds for score estimation across general distributions, and establishes cryptographic lower bounds for score estimation in general Gaussian mixture models, conceptually recovering Song's result and advancing his key open problem.

Abstract

Score estimation is the backbone of score-based generative models (SGMs), especially denoising diffusion probabilistic models (DDPMs). A key result in this area shows that with accurate score estimates, SGMs can efficiently generate samples from any realistic data distribution (Chen et al., ICLR'23; Lee et al., ALT'23). This distribution learning result, where the learned distribution is implicitly that of the sampler's output, does not explain how score estimation relates to classical tasks of parameter and density estimation. This paper introduces a framework that reduces score estimation to these two tasks, with various implications for statistical and computational learning theory: Parameter Estimation: Koehler et al. (ICLR'23) demonstrate that a score-matching variant is statistically inefficient for the parametric estimation of multimodal densities common in practice. In contrast, we show that under mild conditions, denoising score-matching in DDPMs is asymptotically efficient. Density Estimation: By linking generation to score estimation, we lift existing score estimation guarantees to -PAC density estimation, i.e., a function approximating the target log-density within on all but a -fraction of the space. We provide (i) minimax rates for density estimation over Hölder classes and (ii) a quasi-polynomial PAC density estimation algorithm for the classical Gaussian location mixture model, building on and addressing an open problem from Gatmiry et al. (arXiv'24). Lower Bounds for Score Estimation: Our framework offers the first principled method to prove computational lower bounds for score estimation across general distributions. As an application, we establish cryptographic lower bounds for score estimation in general Gaussian mixture models, conceptually recovering Song's (NeurIPS'24) result and advancing his key open problem.

Paper Structure

This paper contains 79 sections, 29 theorems, 136 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Framework and first main tool
  3. Likelihood identity
  4. Applications to parameter estimation
  5. DDPM is an asymptotically efficient parameter estimator
  6. Applications to density estimation
  7. PAC density estimation
  8. Minimax optimal density estimation over Hölder classes
  9. Algorithms for PAC density estimation via score matching
  10. Applications to lower bounds for score estimation
  11. Cryptographic lower bounds for score estimation
  12. Second main tool: Reduction from score estimation to density estimation
  13. Other related work
  14. Score estimation for generation.
  15. Score estimation for parameter recovery.
  16. ...and 64 more sections

Key Result

Lemma 1

Let $P$ be a continuous density over $\mathbb{R}^d$ with finite second moment. Then, for all $x_0\in\mathbb{R}^d$,

Figures (2)

  • Figure 1: Landscape of reductions between score estimation and distribution learning (see \ref{['sec:intro:pacDensity']} for a definition of PAC density estimation). Prior to our work, the only known reduction was from generation to score estimation. Our work shows that score estimation has strong implications for parameter recovery and density estimation. We introduce the notion of DDPM score estimation in \ref{['def:ddpmMain']} and provide relevant background in \ref{['appendix:ddpm']}.
  • Figure 2: Illustration of reductions between different cryptographic problems, density estimation, PAC density estimation, and score estimation. We use these reductions to obtain the hardness of GMM score estimation (\ref{['infthm:scoreCrypto', 'thm:cryptoHardnessGMM']}).

Theorems & Definitions (85)

  • Lemma 1: Likelihood identity
  • Definition 1
  • Definition 2: Evaluation oracle
  • Definition 3: PAC density estimation algorithm
  • Definition 4: Score estimation oracle; informal, see \ref{['def:scoreEstimation']}
  • Lemma 2: Lipschitz score implies sub-Gaussian score
  • proof : Proof of \ref{['lem:identity']}
  • Definition 5: Score estimation oracle
  • Definition 6: Integrated score estimation oracle
  • Lemma 3: Sub-Gaussianity of the score
  • ...and 75 more