Rethinking Global Text Conditioning in Diffusion Transformers

TL;DR

The paper questions the necessity of global text conditioning via pooled embeddings in diffusion transformers and introduces a training-free modulation-guidance framework that reuses the pooled embedding as a controllable signal in the modulation space. By perturbing the global conditioning vector with positive/negative prompt directions and employing dynamic, layer-wise strategies, the method improves image aesthetics, complexity, and editing strength across text-to-image, text-to-video, and instruction-guided editing tasks, with negligible overhead. The approach remains compatible with both CFG-based and CFG-free pipelines and is demonstrated to generalize across multiple state-of-the-art diffusion models. Overall, modulation guidance offers a practical, scalable mechanism to enhance generation quality and controllability without retraining, broadening the utility of diffusion transformers.

Abstract

Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance, suggesting that attention alone is generally sufficient for faithfully propagating prompt information. However, we reveal that the pooled embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various diffusion models, bringing improvements across diverse tasks, including text-to-image/video generation and image editing.

Figures (27)

• Figure 1: (top) Difference between images (DreamSim) with and without CLIP as a function of prompt length. (bot) For long prompts, images without CLIP generally do not differ from the initial ones.
• Figure 2: The modulation guidance enables local (top) and global (bottom) changes and encourages its use to shift a DM toward modes with better properties.
• Figure 3: Analysis on dynamic modulation guidance. (a) Dynamic guidance offers a better trade-off between aesthetic quality and prompt correspondence than constant modulation guidance. (b) We use a step function, controlled by $i$, that skips the first few layers of the model as our form of dynamic guidance. Additional variants are explored in Appendix \ref{['app:strats_dynamic']}.
• Figure 4: After applying modulation guidance, the model focuses more on the desired features, such as hands (a, b).
• Figure 5: Qualitative results of modulation guidance for Aesthetics (top) and Complexity (bottom). The Aesthetics guidance notably improves image quality, while the Complexity guidance can enhance the complexity of both the main object and background details.