Classifier-Free Guidance is a Predictor-Corrector

TL;DR

This paper analyzes classifier-free guidance (CFG) and reveals that CFG interacts differently with DDPM and DDIM, and does not implement the gamma-powered forward process. It reframes CFG as a predictor-corrector method (PCG), proving that in the continuous-time limit CFG is equivalent to a DDIM predictor plus a Langevin corrector on a gamma-powered distribution, with a specific parameter mapping gamma' = 2gamma - 1. Through counterexamples, the authors debunk common myths about CFG and provide a principled explanation of why CFG improves sample quality and prompt adherence. They further demonstrate PCG on Stable Diffusion XL and discuss a broader design space for guided samplers, along with open questions and potential practical improvements.

Abstract

We investigate the theoretical foundations of classifier-free guidance (CFG). CFG is the dominant method of conditional sampling for text-to-image diffusion models, yet unlike other aspects of diffusion, it remains on shaky theoretical footing. In this paper, we disprove common misconceptions, by showing that CFG interacts differently with DDPM (Ho et al., 2020) and DDIM (Song et al., 2021), and neither sampler with CFG generates the gamma-powered distribution $p(x|c)^γp(x)^{1-γ}$. Then, we clarify the behavior of CFG by showing that it is a kind of predictor-corrector method (Song et al., 2020) that alternates between denoising and sharpening, which we call predictor-corrector guidance (PCG). We prove that in the SDE limit, CFG is actually equivalent to combining a DDIM predictor for the conditional distribution together with a Langevin dynamics corrector for a gamma-powered distribution (with a carefully chosen gamma). Our work thus provides a lens to theoretically understand CFG by embedding it in a broader design space of principled sampling methods.

Figures (7)

• Figure 1: CFG vs. PCG. We prove that the DDPM variant of classifier-free guidance (top) is equivalent to a kind of predictor-corrector method (bottom), in the continuous limit. We call this latter method "predictor-corrector guidance" (PCG), defined in Section \ref{['sec:pcg_warmup']}. The equivalence holds for all CFG guidance strengths $\gamma$, with corresponding PCG parameter $\gamma'=(2\gamma-1)$, as given in Theorem \ref{['thm:main']}. Samples from SDXL with prompt: "photograph of a cat eating sushi using chopsticks".
• Figure 2: Counterexamples: $\textsf{CFG}_\textsf{DDIM} \neq \textsf{CFG}_\textsf{DDPM} \neq$ gamma-powered.$\textsf{CFG}_\textsf{DDIM}$ and $\textsf{CFG}_\textsf{DDPM}$ do not generate the same output distribution, even when using the same score function. Moreover, neither generated distribution is the gamma-powered distribution $p_{0,\gamma}(x|c)$. (Left) Counterexample 1 (section \ref{['sec:counterex1']}): $\textsf{CFG}_\textsf{DDIM}$ yields a sharper distribution than $\textsf{CFG}_\textsf{DDPM}$, and both are sharper than $p_{0,\gamma}(x|c)$. (Right) Counterexample 2 (section \ref{['sec:counterex2']}): Neither $\textsf{CFG}_\textsf{DDIM}$ nor $\textsf{CFG}_\textsf{DDPM}$ yield even a scaled version of the gamma-powered distribution $p_{0,\gamma}(x|c) = \mathcal{N}(-3,1)$. The $\textsf{CFG}_\textsf{DDPM}$ distribution is mean-shifted relative to $p_{0,\gamma}(x|c)$. The $\textsf{CFG}_\textsf{DDIM}$ distribution is mean-shifted and not even Gaussian (note the asymmetrical shape).
• Figure 3: CFG is equivalent to PCG for particular parameter choices.
• Figure 4: An example where guidance benefits generalization. Suppose that the conditional distribution for $c=0$ is a a GMM with a dominant cluster, as shown in purple, and the unconditional distribution is uniform (details in Appendix \ref{['app:prob_4']}). We sample with DDPM using exact scores vs. scores learned by training a small MLP with early stopping. The scores are learned more accurately near the dominant cluster. (Left) For conditional sampling (no guidance), DDPM is expected to sample from the conditional distribution (purple curve). However, DDPM-with-learned-scores (orange) samples less accurately than DDPM-with-exact-scores (blue) away from the dominant cluster (where the learned scores are inaccurate) (note the prevalence of blue samples in low-probability regions). (Center) With guidance $\gamma=3$, $p_{0,\gamma}(x|c=0)$ (red) and both samplers concentrate around the dominant cluster (where the learned scores are accurate), reducing the generalization gap between the learned and exact models. (Right) Exact vs. learned condition scores $\nabla \log p(x|c=0)$.
• Figure 5: Effect of Guidance and Correction. Each grid shows SDXL samples using $\textsf{PCG}_\textsf{DDIM}$, as the guidance strength $\gamma$ and Langevin iterations $K$ are varied. Left: "photograph of a dog drinking coffee with his friends". Right: "a tree reflected in the hood of a blue car". (Zoom in to view).

Theorems & Definitions (4)

• Theorem 1: DDIM $\ne$ DDPM; informal
• Theorem 2: CFG $\neq$ gamma-sharpening, informal
• Theorem 3: CFG is predictor-corrector
• proof