Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis

TL;DR

The paper tackles semantic misalignment in diffusion-based text-to-image synthesis by introducing Noise Diffusion (ND), a training-free framework that guides the optimization of the initial latent via LVLM supervision using a Visual Question Answering score. ND updates the latent through a forward-diffusion step and selects gradient-guided noise to improve semantic faithfulness while preserving latent distribution, backed by a theoretical condition for score improvement. Empirically, ND consistently enhances semantic alignment across multiple diffusion backbones and LVLMs, outperforming prior latent-optimization approaches like InitNo in both simple and complex prompts. The approach is modular and plug-and-play, providing a practical path to more faithful text-to-image generation with publicly available code.

Abstract

Diffusion models have achieved impressive success in generating photorealistic images, but challenges remain in ensuring precise semantic alignment with input prompts. Optimizing the initial noisy latent offers a more efficient alternative to modifying model architectures or prompt engineering for improving semantic alignment. A latest approach, InitNo, refines the initial noisy latent by leveraging attention maps; however, these maps capture only limited information, and the effectiveness of InitNo is highly dependent on the initial starting point, as it tends to converge on a local optimum near this point. To this end, this paper proposes leveraging the language comprehension capabilities of large vision-language models (LVLMs) to guide the optimization of the initial noisy latent, and introduces the Noise Diffusion process, which updates the noisy latent to generate semantically faithful images while preserving distribution consistency. Furthermore, we provide a theoretical analysis of the condition under which the update improves semantic faithfulness. Experimental results demonstrate the effectiveness and adaptability of our framework, consistently enhancing semantic alignment across various diffusion models. The code is available at https://github.com/Bomingmiao/NoiseDiffusion.

Figures (15)

• Figure 1: Example results of Stable Diffusion models and ours. Given a fixed initial noisy latent, we optimize the latent toward an area that can generate images aligned with the input prompts.
• Figure 2: The framework of Noise Diffusion. The image generated from the initial noisy latent is fed into the LVLM along with a question formatted as, "Does this figure show '{prompt}'? Please answer yes or no." The probability of the token "Yes" serves as the VQA score. The step size for updating the noisy latent is dynamically adjusted based on the score value. Gradient information is then used to select the optimal noise for the update according to the step difference.
• Figure 3: Qualitative comparison for simple cases. Each image is generated with the same prompt and random seed for all methods. The images generated by our method contain objects that most closely match the features described in the prompts.
• Figure 4: Qualitative comparison for complex cases. Each image is generated with the same text prompt and random seed for all methods. Our method exhibits strong understanding of the positional relationships described in the prompts.
• Figure 5: The average CLIP score and VQA score of the generated images both increase as the epochs progress and eventually converge. Compared to InitNo, the Noise Diffusion (ND) method consistently outperforms InitNo in both CLIP and VQA scores.