Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Score-based sampling without diffusions: Guidance from a simple and modular scheme

M. J. Wainwright

TL;DR

A modular scheme that reduces score-based sampling to solving a short sequence of ``nice''sampling problems, for which high-accuracy samplers are known, and establishes novel guarantees for both uni-modal and multi-modal densities.

Abstract

Sampling based on score diffusions has led to striking empirical results, and has attracted considerable attention from various research communities. It depends on availability of (approximate) Stein score functions for various levels of additive noise. We describe and analyze a modular scheme that reduces score-based sampling to solving a short sequence of ``nice'' sampling problems, for which high-accuracy samplers are known. We show how to design forward trajectories such that both (a) the terminal distribution, and (b) each of the backward conditional distribution is defined by a strongly log concave (SLC) distribution. This modular reduction allows us to exploit \emph{any} SLC sampling algorithm in order to traverse the backwards path, and we establish novel guarantees with short proofs for both uni-modal and multi-modal densities. The use of high-accuracy routines yields $\varepsilon$-accurate answers, in either KL or Wasserstein distances, with polynomial dependence on $\log(1/\varepsilon)$ and $\sqrt{d}$ dependence on the dimension.

Score-based sampling without diffusions: Guidance from a simple and modular scheme

TL;DR

A modular scheme that reduces score-based sampling to solving a short sequence of ``nice''sampling problems, for which high-accuracy samplers are known, and establishes novel guarantees for both uni-modal and multi-modal densities.

Abstract

Sampling based on score diffusions has led to striking empirical results, and has attracted considerable attention from various research communities. It depends on availability of (approximate) Stein score functions for various levels of additive noise. We describe and analyze a modular scheme that reduces score-based sampling to solving a short sequence of ``nice'' sampling problems, for which high-accuracy samplers are known. We show how to design forward trajectories such that both (a) the terminal distribution, and (b) each of the backward conditional distribution is defined by a strongly log concave (SLC) distribution. This modular reduction allows us to exploit \emph{any} SLC sampling algorithm in order to traverse the backwards path, and we establish novel guarantees with short proofs for both uni-modal and multi-modal densities. The use of high-accuracy routines yields -accurate answers, in either KL or Wasserstein distances, with polynomial dependence on and dependence on the dimension.
Paper Structure (61 sections, 11 theorems, 89 equations, 1 figure)

This paper contains 61 sections, 11 theorems, 89 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Score-based diffusions:
  3. High-accuracy sampling for "nice" distributions:
  4. Unifying yields a reduction:
  5. A geometric view
  6. Related work
  7. Paper organization:
  8. Background and set-up
  9. Fast samplers for SLC distributions
  10. SLC classes:
  11. Annealed Stein scores
  12. Second-order Tweedie and Hessian structure
  13. Exponential acceleration for log-concave sampling
  14. Procedure and its guarantee
  15. Choice of $s_k$:
  16. ...and 46 more sections

Key Result

Lemma 1

We have the second-order Tweedie formula

Figures (1)

  • Figure 1: Contour plots of a 2-$D$ probability densities, along with quiver plots of the Stein scores. (a) Forward marginal distributions $p_k \equiv p_{Y_k}$ for the forward process $(Y_1, Y_2, Y_3, Y_4, Y_5)$: they are initially multi-modal, but then become progressively simpler. It is straightforward to draw samples from the density $p_{Y_5}$ in the fifth panel. (b) Backward conditional distributions $p_{k \mid k+1} \equiv p_{Y_k \mid Y_{k+1}}$ for $k = 1, 2, 3, 4$, obtained with one random choice of sequence $Y = (Y_2, \ldots, Y_5)$ defining the backward conditionals $p_{k \mid k+1}(\cdot | Y_{k+1})$. Panels (c) and (d) give same illustration with different choices of backward sequence. All three panels (b)--(d) show "nice" backward distributions; our theory gives conditions under which each of these backward conditional distributions are in the SLC class.

Theorems & Definitions (12)

  • Lemma 1: Forward and conditional Hessians
  • Theorem 1: Logarithmic reduction to SLC black box sampling
  • Lemma 2: Properties of forward trajectory
  • Lemma 3: Error propagation in backwards kernel $\mathscr{B}_k$
  • Lemma 4: Propagation of spectral control
  • Theorem 2: SLC reduction for multi-modal case
  • Corollary 1
  • Lemma 5: Trajectory control for multi-modal case
  • Lemma 6: Multi-modal error propagation
  • Lemma 7: Hessian identity for marginal log densities
  • ...and 2 more