Highly Efficient Test-Time Scaling for T2I Diffusion Models with Text Embedding Perturbation

TL;DR

<3-5 sentence high-level summary>: This work tackles the limited exploration of randomness in test-time scaling for text-to-image diffusion models. It introduces Text Embedding Perturbation (TEP), a step-based, frequency-guided perturbation of text embeddings that complements spatial noise and SDE-based randomness. Frequency-domain analysis reveals that low-frequency spatial noise benefits early denoising whereas high-frequency embedding perturbations refine details later, guiding the design of a two-fold framework with adaptive perturbation across time, embedding components, and cross-attention layers. TEP is plug-and-play across existing TTS methods and backbones, delivering significant performance gains with negligible computational overhead on multiple benchmarks and reward metrics. This approach offers a practical path to boost generative diversity and fidelity in T2I diffusion models without retraining.

Abstract

Test-time scaling (TTS) aims to achieve better results by increasing random sampling and evaluating samples based on rules and metrics. However, in text-to-image(T2I) diffusion models, most related works focus on search strategies and reward models, yet the impact of the stochastic characteristic of noise in T2I diffusion models on the method's performance remains unexplored. In this work, we analyze the effects of randomness in T2I diffusion models and explore a new format of randomness for TTS: text embedding perturbation, which couples with existing randomness like SDE-injected noise to enhance generative diversity and quality. We start with a frequency-domain analysis of these formats of randomness and their impact on generation, and find that these two randomness exhibit complementary behavior in the frequency domain: spatial noise favors low-frequency components (early steps), while text embedding perturbation enhances high-frequency details (later steps), thereby compensating for the potential limitations of spatial noise randomness in high-frequency manipulation. Concurrently, text embedding demonstrates varying levels of tolerance to perturbation across different dimensions of the generation process. Specifically, our method consists of two key designs: (1) Introducing step-based text embedding perturbation, combining frequency-guided noise schedules with spatial noise perturbation. (2) Adapting the perturbation intensity selectively based on their frequency-specific contributions to generation and tolerance to perturbation. Our approach can be seamlessly integrated into existing TTS methods and demonstrates significant improvements on multiple benchmarks with almost no additional computation. Code is available at \href{https://github.com/xuhang07/TEP-Diffusion}{https://github.com/xuhang07/TEP-Diffusion}.

Figures (9)

• Figure 1: Top: Comparison of text embedding perturbation with previous randomness. Bottom: The corresponding generated images of the Top. Our method is plug-and-play.
• Figure 2: In (a), we demonstrate that text embedding perturbation consistently enhances generation diversity across all CFG scales, highlighting its potential as a novel format of randomness. However, in (b), we reveal that existing approaches(CADS) incorporating text embedding perturbation for better generative diversity are incompatible with TTS in T2I diffusion models, as they may not maintain image quality. We use SD3.5 with ImageReward for evaluations, and results of more backbones are in Appendix.
• Figure 3: In (a), we switch from SDE to ODE at specific steps. We observe that the noise injected by the SDE in the early steps helps select better-quality images, while in the later steps, it has a negative impact. In (b), we attenuate the SDE-injected noise in the frequency domain at specific steps to analyze the influence of its high- and low-frequency components on generation. We find that the low-frequency one play a crucial role throughout the entire process, whereas suppressing the high-frequency components improves image quality in TTS. We use SD3.5 with ImageReward for evaluations, and more backbones' results are in Appendix.
• Figure 4: We demonstrate the differences and complementarity between text embedding perturbation and spatial noise randomness in TTS. In (a), we only introduced randomness at fixed timesteps while keeping other steps unchanged, and measured the MSE between images generated with and without randomness. Spatial noise randomness has a greater impact during the low-frequency generation phase (early steps), while text embedding perturbation plays a more significant role in the high-frequency phase (late steps). Furthermore, when both randomness formats are combined, the variation in images shows a more pronounced improvement. In (b), we further provide visual results to support the previous discussion. We use SD3.5 with ImageReward for evaluations, and results of more backbones are in Appendix.
• Figure 5: (a) shows text embedding perturbation should be applied discriminatively across three dimensions: timesteps, the specific components of the embedding, and the depth of Cross-Attention layers, which are detailed in (b), (c), and (d), respectively. We perform perturbation across each of the three dimensions and evaluate rewards with BoN. Specifically: (b) illustrates that perturbation at early timesteps yields minimal improvement, while perturbation in later timesteps provides a significant boost. (c) indicates that the unconditional text embedding exhibits greater tolerance for perturbation compared to the conditional text embedding. (d) shows that shallower layers demonstrate higher tolerance for perturbation than deeper layers. We use SD3.5 with ImageReward for evaluations, and more backbones' results are in Appendix.