Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models

TL;DR

S$^2$-Guidance is proposed, a novel method that leverages stochastic block-dropping during the forward process to construct stochastic sub-networks, effectively guiding the model away from potential low-quality predictions and toward high-quality outputs.

Abstract

Classifier-free Guidance (CFG) is a widely used technique in modern diffusion models for enhancing sample quality and prompt adherence. However, through an empirical analysis on Gaussian mixture modeling with a closed-form solution, we observe a discrepancy between the suboptimal results produced by CFG and the ground truth. The model's excessive reliance on these suboptimal predictions often leads to semantic incoherence and low-quality outputs. To address this issue, we first empirically demonstrate that the model's suboptimal predictions can be effectively refined using sub-networks of the model itself. Building on this insight, we propose S$^2$-Guidance, a novel method that leverages stochastic block-dropping during the forward process to construct stochastic sub-networks, effectively guiding the model away from potential low-quality predictions and toward high-quality outputs. Extensive qualitative and quantitative experiments on text-to-image and text-to-video generation tasks demonstrate that S$^2$-Guidance delivers superior performance, consistently surpassing CFG and other advanced guidance strategies. Our code will be released.

Figures (16)

• Figure 1: Visual results of $S^2$-Guidance versus CFG. Our proposed method $S^2$-Guidance significantly elevates the quality and coherence of both T2I and T2V generation. Observe (in examples surrounding the center): Our method produces generations with superior temporal dynamics, including more pronounced motion (bear) and dynamic camera angles that convey speed (car). It renders finer details, such as the astronaut's transparent helmet and rich facial details, and creates images with fewer artifacts (runner, woman with umbrella), richer artistic detail (abstract portrait, castle, colored powder exploding), and improved object coherence (cat and rocket, sheep). See Appendix \ref{['app:figure1_prompts']} for our prompts.
• Figure 2: An illustration of our guidance mechanism on the generation quality manifold. Unlike suboptimal CFG guidance (gray), $S^2$-Guidance derives a corrective signal (blue) via stochastic block-dropping, steering the generation update (purple) toward the optimal quality peak (yellow).
• Figure 3: $S^2$-Guidance successfully balances guidance strength and distribution fidelity. Comparison on 1D (top) and 2D (bottom) toy examples. Unlike CFG, which distorts the sample distribution (see red boxes), or other methods that fail to separate modes, $S^2$-Guidance accurately captures both the location and shape of the ground truth distributions (semi-transparent).
• Figure 4: $S^2$-Guidance avoids the distributional collapse of CFG on CIFAR-10. t-SNE shows generated features (points) vs. real data (contours). CFG (b) exhibits severe collapse, whereas $S^2$-Guidance (e) preserves the distribution's structure while ensuring class separation. See (f) for qualitative examples.
• Figure 5: $S^2$-Guidance consistently generates superior images in both aesthetic quality and prompt coherence. While existing guidance methods often produce artifacts, distorted objects, or fail to follow complex prompts (see red boxes), our approach yields clean, coherent, and visually pleasing results without such flaws.