Towards Understanding the Mechanisms of Classifier-Free Guidance

TL;DR

This work addresses the unclear mechanisms behind classifier-free guidance (CFG) in diffusion models by analyzing CFG within an optimal linear diffusion framework for Gaussian data. It derives a decomposition of CFG into three components: a mean-shift term toward the class mean, a positive CPC term that amplifies class-specific features, and a negative CPC term that suppresses features common to the unconditional data, with the CPC directions obtained from the difference of conditional and unconditional posteriors. The authors show that linear CFG closely mirrors nonlinear CFG at high-to-moderate noise and remains informative in the nonlinear regime via an adaptive, Jacobian-based CPC interpretation, thereby illuminating CFG's operating principles and its effect on sample quality and class separation. The findings offer practical guidance for designing training objectives to encourage class-specific covariance structures and point to CPCA-based avenues for more controllable and interpretable diffusion-based generation, including extensions to Gaussian mixture data.

Abstract

Classifier-free guidance (CFG) is a core technique powering state-of-the-art image generation systems, yet its underlying mechanisms remain poorly understood. In this work, we begin by analyzing CFG in a simplified linear diffusion model, where we show its behavior closely resembles that observed in the nonlinear case. Our analysis reveals that linear CFG improves generation quality via three distinct components: (i) a mean-shift term that approximately steers samples in the direction of class means, (ii) a positive Contrastive Principal Components (CPC) term that amplifies class-specific features, and (iii) a negative CPC term that suppresses generic features prevalent in unconditional data. We then verify these insights in real-world, nonlinear diffusion models: over a broad range of noise levels, linear CFG resembles the behavior of its nonlinear counterpart. Although the two eventually diverge at low noise levels, we discuss how the insights from the linear analysis still shed light on the CFG's mechanism in the nonlinear regime.

Figures (39)

• Figure 1: Comparison of Sampling Trajectories. For high to moderate noise levels ($\sigma(t)\in(4,80]$), the linear denoisers well approximate the learned deep denoisers. Though the two models diverge in lower noise reigmes, their final samples still match in overall structure.
• Figure 2: Effects of CFG. Left and right compare naive conditional sampling (top rows) versus CFG-guided sampling (bottom rows) for deep diffusion models (EDM) and linear Gaussian diffusion models, respectively. Each grid ceil corresponds to the same initial noise. While naive conditional samples lack class-specific clarity, CFG significantly improves both visual quality and distinctiveness. The conditional linear models are built with class-specific means and covariances. Please refer to \ref{['sec: more discussion on covariance structure']} for more experiment results.
• Figure 3: Class‑to‑Class Similarity. Each cell reports the FID between datasets of two classes, built with (i) training data (ii) data generated by naive conditional sampling and (iii) data generated by CFG sampling (refer to \ref{['subsec: Quantitative Results']} for experiment details and more results.)
• Figure 4: Distinct effects of different CFG components. (a) CFG substantially enhances class-specific features (in both EDM and linear diffusion). (b) Top row: PCs, positive/negative CPCs, and $\boldsymbol\mu_c - \boldsymbol\mu_{uc}$. Bottom row: generated samples when each component is applied in isolation. (c) One‑dimensional densities of generated samples after projection onto key directions. The left column corresponds to the linear diffusion model, whereas the right column corresponds to the EDM model. Top row: project onto leading positive CPC. Middle row: project onto negative CPC. Third row: project onto the mean-shift direction. Here we only plot the resulting histograms for the first positive and negative CPCs but the same patterns hold for subsequent CPCs. For experimental details and more results, please refer to \ref{['Distinct Effects of the CFG Components appendix']}.
• Figure 5: Linear-to-nonlinear transition in diffusion models. (a) and (b) compare nonlinear CFG and linear CFG applied to a deep diffusion model (EDM). The leftmost column shows unguided samples; subsequent columns show final samples when guidance is applied only at a specific noise level, with $\gamma=15$ (See \ref{['fig:linear_nonlinear_transition_extra']} for more examples).

Theorems & Definitions (2)

• Theorem 1
• Theorem 2