Entropy Rectifying Guidance for Diffusion and Flow Models

TL;DR

Entropy Rectifying Guidance (ERG) introduces an inference-time, attention-energy Based mechanism to rectify the energy landscape of attention in diffusion transformers. By temperature-scaling in both image denoisers and text encoders and formulating a controllable energy function with hyperparameters, ERG produces a weaker predictive signal that guides generation toward higher quality, greater diversity, and stronger prompt consistency, without requiring extra models. ERG demonstrates significant improvements across text-to-image, class-conditional and unconditional generation and integrates smoothly with CADS and APG, achieving favorable Pareto fronts with minimal additional computation. The approach, grounded in Hopfield energy interpretations of attention, offers a general, architecture-friendly guidance paradigm for diffusion/flow models with broad practical impact.

Abstract

Guidance techniques are commonly used in diffusion and flow models to improve image quality and input consistency for conditional generative tasks such as class-conditional and text-to-image generation. In particular, classifier-free guidance (CFG) is the most widely adopted guidance technique. It results, however, in trade-offs across quality, diversity and consistency: improving some at the expense of others. While recent work has shown that it is possible to disentangle these factors to some extent, such methods come with an overhead of requiring an additional (weaker) model, or require more forward passes per sampling step. In this paper, we propose Entropy Rectifying Guidance (ERG), a simple and effective guidance method based on inference-time changes in the attention mechanism of state-of-the-art diffusion transformer architectures, which allows for simultaneous improvements over image quality, diversity and prompt consistency. ERG is more general than CFG and similar guidance techniques, as it extends to unconditional sampling. We show that ERG results in significant improvements in various tasks, including text-to-image, class-conditional and unconditional image generation. We also show that ERG can be seamlessly combined with other recent guidance methods such as CADS and APG, further improving generation results.

Figures (20)

• Figure 1: Qualitative comparison of classifier-free guidance (CFG) and our Entropy Rectifying Guidance (ERG). The images generated using ERG (bottom rows) exhibit greater quality and diversity than standard CFG. Images are generated using 50 Euler steps; each column corresponds to a different random seed for the generations.
• Figure 2: Pareto fronts on consistency-diversity-quality for text-to-image generation. Comparing ERG + APG (dots) with APG (crosses). We sweep over different guidance scales (each marked with a different color), and hyper-parameters $\alpha, \gamma, \tau$ for ERG. Dashed lines trace the Pareto fronts for each plot. We measure consistency with VQAScore, quality with density and diversity with coverage.
• Figure 3: Unconditional generation results. Compared to not using guidance (top), our ERG generates more realistic and detailed images and more coherent structure (bottom). Images obtained from T2I model at $512$ with empty prompt as input. Samples in each column use the same seed.
• Figure 4: Temperature rescaling in the conditioning for text-to-image generation (C-ERG). We vary the text encoder's attention temperatures $\tau_c$. Each curve corresponds to a different guidance strength $w$. The left-most point on each curve represents the result for standard CFG.
• Figure 5: Effect of C-ERG on initial velocity step. Standard CFG shows very localized initial marginal variance while the conditional variance is much larger. This means that the negative model predicts an initial velocity that is very similar when varying the initial noise (and prompt in case o C-ERG), and is largely decorrelated from the conditional prediction, leading to a lack of diversity in the generated samples. Conversely, C-ERG results in much higher marginal variance and smaller conditional variance, reducing the error accumulation that can happen at earlier timesteps and leading to better diversity.