Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PiCo: Enhancing Text-Image Alignment with Improved Noise Selection and Precise Mask Control in Diffusion Models

Chang Xie, Chenyi Zhuang, Pan Gao

TL;DR

PiCo addresses misalignment in text-to-image diffusion by two training-free components: fast noise selection that ranks initial noises via ITM and concept scores, and a referring mask control that modulates cross-attention with pixel-level masks. The approach improves attribute binding and object attendance for complex prompts while remaining efficient due to a small $T_s$ and early denoising intervention at $T_c$. Extensive experiments on CC-500 and T2I-Compbench demonstrate higher alignment and image quality than baselines, with ablations highlighting the importance of both noise scoring and precise mask control. The work offers a practical, model-agnostic route to sharper text-image correspondence and suggests avenues for adaptive hyperparameters and addressing CLIPSeg biases.

Abstract

Advanced diffusion models have made notable progress in text-to-image compositional generation. However, it is still a challenge for existing models to achieve text-image alignment when confronted with complex text prompts. In this work, we highlight two factors that affect this alignment: the quality of the randomly initialized noise and the reliability of the generated controlling mask. We then propose PiCo (Pick-and-Control), a novel training-free approach with two key components to tackle these two factors. First, we develop a noise selection module to assess the quality of the random noise and determine whether the noise is suitable for the target text. A fast sampling strategy is utilized to ensure efficiency in the noise selection stage. Second, we introduce a referring mask module to generate pixel-level masks and to precisely modulate the cross-attention maps. The referring mask is applied to the standard diffusion process to guide the reasonable interaction between text and image features. Extensive experiments have been conducted to verify the effectiveness of PiCo in liberating users from the tedious process of random generation and in enhancing the text-image alignment for diverse text descriptions.

PiCo: Enhancing Text-Image Alignment with Improved Noise Selection and Precise Mask Control in Diffusion Models

TL;DR

PiCo addresses misalignment in text-to-image diffusion by two training-free components: fast noise selection that ranks initial noises via ITM and concept scores, and a referring mask control that modulates cross-attention with pixel-level masks. The approach improves attribute binding and object attendance for complex prompts while remaining efficient due to a small and early denoising intervention at . Extensive experiments on CC-500 and T2I-Compbench demonstrate higher alignment and image quality than baselines, with ablations highlighting the importance of both noise scoring and precise mask control. The work offers a practical, model-agnostic route to sharper text-image correspondence and suggests avenues for adaptive hyperparameters and addressing CLIPSeg biases.

Abstract

Advanced diffusion models have made notable progress in text-to-image compositional generation. However, it is still a challenge for existing models to achieve text-image alignment when confronted with complex text prompts. In this work, we highlight two factors that affect this alignment: the quality of the randomly initialized noise and the reliability of the generated controlling mask. We then propose PiCo (Pick-and-Control), a novel training-free approach with two key components to tackle these two factors. First, we develop a noise selection module to assess the quality of the random noise and determine whether the noise is suitable for the target text. A fast sampling strategy is utilized to ensure efficiency in the noise selection stage. Second, we introduce a referring mask module to generate pixel-level masks and to precisely modulate the cross-attention maps. The referring mask is applied to the standard diffusion process to guide the reasonable interaction between text and image features. Extensive experiments have been conducted to verify the effectiveness of PiCo in liberating users from the tedious process of random generation and in enhancing the text-image alignment for diverse text descriptions.
Paper Structure (17 sections, 15 equations, 18 figures, 6 tables)

This paper contains 17 sections, 15 equations, 18 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Fast Noise Selection
  5. Referring Mask Control
  6. Experiments
  7. Experimental setup
  8. Comparison Results
  9. Ablation Study
  10. Conclusion
  11. Preliminaries
  12. Evaluation Metrics Discussion
  13. More Implementation Details
  14. Additional Ablation Study
  15. Limitation
  16. ...and 2 more sections

Figures (18)

  • Figure 1: Overview of the proposed PiCo. (1) The noise selection module fastly assesses the quality of the random noise through the ITM score and concept scores. (2) The referring mask control module intervenes in the cross-attention layer with pixel-level concept and exclusive masks.
  • Figure 2: Qualitative comparison using prompts from T2I-Compbench. Each case uses the same initialized noise, which is output by the noise selection module.
  • Figure 3: Ablation study on $T_c$ to stop the mask control. Under a small value, the intervention is not enough but a larger one can lead to degeneration.
  • Figure 4: Ablation study on $\gamma$. A small value of $\gamma$ can not well disentangle different concepts, while a large value causes artifacts in the generated image (best viewed zoomed in). We empirically set $\gamma = 15$.
  • Figure 5: A screenshot of the human evaluation of text-image alignment. Evaluators are presented with two images by PiCo and another method of baselines to choose which one better reflects the target prompt or "tie".
  • ...and 13 more figures