Emergence of Distortions in High-Dimensional Guided Diffusion Models

TL;DR

The paper investigates distortions introduced by classifier-free guidance (CFG) in high-dimensional diffusion models, formalizing distortion as the mismatch between CFG-induced and true conditional distributions. It develops a theoretical framework combining exact Gaussian targets, dynamical mean-field theory, and random energy model (REM) analysis to predict when CFG distorts the target distribution, showing distortions persist when the number of classes grows exponentially with dimension and vanish in the sub-exponential regime. Vanilla CFG is shown to expand the conditional mean and shrink the variance, reducing sample diversity, and a negative-guidance window scheduling is proposed to recover diversity while preserving class separability. The work integrates real-data experiments with Gaussian-mixture analyses and provides phase-diagram guidance for CFG scheduling, offering a principled path to mitigate diversity loss in high-dimensional conditional generation.

Abstract

Classifier-free guidance (CFG) is the de facto standard for conditional sampling in diffusion models, yet it often leads to a loss of diversity in generated samples. We formalize this phenomenon as generative distortion, defined as the mismatch between the CFG-induced sampling distribution and the true conditional distribution. Considering Gaussian mixtures and their exact scores, and leveraging tools from statistical physics, we characterize the onset of distortion in a high-dimensional regime as a function of the number of classes. Our analysis reveals that distortions emerge through a phase transition in the effective potential governing the guided dynamics. In particular, our dynamical mean-field analysis shows that distortion persists when the number of modes grows exponentially with dimension, but vanishes in the sub-exponential regime. Consistent with prior finite-dimensional results, we further demonstrate that vanilla CFG shifts the mean and shrinks the variance of the conditional distribution. We show that standard CFG schedules are fundamentally incapable of preventing variance shrinkage. Finally, we propose a theoretically motivated guidance schedule featuring a negative-guidance window, which mitigates loss of diversity while preserving class separability.

Figures (16)

• Figure 1: Measures of distortion from a guided Stable Diffusion (v1.5) model, in feature space, as a function of the guidance level $w$, averaged over 100 samples. Blue circles refer to CLIP feature extractor, yellow ones to DINOv2. Left: Quadratic distance between the mean across the features at a given guidance level and the one measured at $w = 0$. The increasing distance signals a gain in class separability. Right: Participation ratio of the eigenvalues of the empirical covariance matrix across samples showing a loss of sample diversity with increasing guidance.
• Figure 2: Samples generated for the prompt a fantasy landscape with castles and dragons, vibrant colors, digital art with Stable Diffusion v1.5. Rows are different random seeds, columns refer to guidance levels.
• Figure 3: The coefficients $\lambda$ and $\Lambda$ governing the distortion of CFG in the Gaussian setting, at $t = 0$ and as a function of $w$ and the ratio $s/r$, where $(s,r)$ are eigenvalues, respectively, of $\Sigma_{x|c}$ and $\Sigma_{xx}$. Since $\lambda \geq 1$, the mean of the conditional target distribution is always expanded, while $\Lambda \leq 1$ implies a systematic contraction of the covariance matrix.
• Figure 4: Measure of distortion from numerical simulations for CFG on jointly Gaussian classes and data, showing increased class separation and decreased diversity with increasing $w$. Dimensions are $d_1 = 1$, $d_2 = 9$. Left: the norm of the CFG mean divided by the true conditional mean. Right: Frobenius norm of $\Sigma_w$ divided by the true conditional covariance matrix.
• Figure 5: Speciation time $t_s$ and the distortion estimators in the exponential regime predicted by the theory as functions of the control parameters $\beta = \log(M)/d$ and $w$, for $\sigma^2 = 0.5$. In the white region, there is no speciation and therefore we have no transition to the conditional phase. This regime displays strong distortion in the conditional sampling as testified by the behavior of $\delta_{\mu}$ and $\delta_{\sigma^2}$. In the small $\beta$ regime instead, where the transition occurs, distortion is weak.