Saddle-Free Guidance: Improved On-Manifold Sampling without Labels or Additional Training

TL;DR

Saddle-Free Guidance (SFG) introduces a training-free, model-agnostic approach for guiding score-based generative models by exploiting the positive curvature in saddle regions of the log-density. It computes the maximal eigenvalue of the score Jacobian via shifted power iteration and augments the score with a curvature-driven term, maintaining a cost similar to classifier-free guidance. Across ImageNet-512 and text-to-image benchmarks, SFG delivers state-of-the-art unconditional and competitive conditional results, and when combined with Auto-Guidance sets new records, all without additional training or labels. The method also extends to diffusion-based text-to-image models, achieving high fidelity and diversity while preserving prompt adherence.

Abstract

Score-based generative models require guidance in order to generate plausible, on-manifold samples. The most popular guidance method, Classifier-Free Guidance (CFG), is only applicable in settings with labeled data and requires training an additional unconditional score-based model. More recently, Auto-Guidance adopts a smaller, less capable version of the original model to guide generation. While each method effectively promotes the fidelity of generated data, each requires labeled data or the training of additional models, making it challenging to guide score-based models when (labeled) training data are not available or training new models is not feasible. We make the surprising discovery that the positive curvature of log density estimates in saddle regions provides strong guidance for score-based models. Motivated by this, we develop saddle-free guidance (SFG) which maintains estimates of maximal positive curvature of the log density to guide individual score-based models. SFG has the same computational cost of classifier-free guidance, does not require additional training, and works with off-the-shelf diffusion and flow matching models. Our experiments indicate that SFG achieves state-of-the-art FID and FD-DINOv2 metrics in single-model unconditional ImageNet-512 generation. When SFG is combined with Auto-Guidance, its unconditional samples achieve general state-of-the-art in FD-DINOv2 score. Our experiments with FLUX.1-dev and Stable Diffusion v3.5 indicate that SFG boosts the diversity of output images compared to CFG while maintaining excellent prompt adherence and image fidelity.

Figures (12)

• Figure 1: SFG improves image realism by steering samples away from saddle regions. SFG combined with Auto-Guidance achieves state-of-the-art FD-DINOv2 on unconditional ImageNet-512.
• Figure 2: Explicit score matching loss (\ref{['eqn:esm_loss']}) on the test set for the $16$-simplex task. Test points within saddle regions experience the highest loss and benefit the least from DiT scaling.
• Figure 3: Comparison of sampling without guidance, with classifier-free guidance, with auto-guidance, and with saddle-free guidance on the toy fractal manifold learning task from karras2024guiding. Sampling without guidance produces many unlikely 'outliers' which are caught in saddle regions between modes. CFG and its variants reduce outlier in saddle regions, but they bias generation and reduce sample diversity. Auto-guidance avoids outliers in saddle regions, but it is overly restrictive and reduces diversity on the manifold. Saddle-free guidance prevents outliers while maintaining sample diversity on the manifold.
• Figure 4: Effect of step size parameter on the FID and FD-DINOv2 tradeoff for EDM2-s-conditional (EMA 0.130).
• Figure 5: Comparison of FD-DINOv2 and FID scores for single-model unconditional guidance methods with EDM2-s on ImageNet-512.

Theorems & Definitions (1)

• Definition 3.1: Saddle Points