VSC: Visual Search Compositional Text-to-Image Diffusion Model

TL;DR

This work tackles the challenge of accurately binding multiple attribute-object pairs in text-to-image diffusion by decomposing prompts into per-pair subprompts, generating reference images for each pair, and constructing visual prototypes that augment text conditioning through a trainable MLP. A segmentation-guided local attention loss aligns cross-attention maps with object regions, addressing misalignment that plagues prior methods. Evaluations on the T2I-CompBench benchmark show state-of-the-art performance across color, texture, and shape attributes, with strong human judgments of image quality, and evidence of data-scale and transferability benefits. The approach remains tuning-free for the diffusion backbone, scales effectively with prompt complexity, and offers practical improvements for compositional image synthesis in real-world applications.

Abstract

Text-to-image diffusion models have shown impressive capabilities in generating realistic visuals from natural-language prompts, yet they often struggle with accurately binding attributes to corresponding objects, especially in prompts containing multiple attribute-object pairs. This challenge primarily arises from the limitations of commonly used text encoders, such as CLIP, which can fail to encode complex linguistic relationships and modifiers effectively. Existing approaches have attempted to mitigate these issues through attention map control during inference and the use of layout information or fine-tuning during training, yet they face performance drops with increased prompt complexity. In this work, we introduce a novel compositional generation method that leverages pairwise image embeddings to improve attribute-object binding. Our approach decomposes complex prompts into sub-prompts, generates corresponding images, and computes visual prototypes that fuse with text embeddings to enhance representation. By applying segmentation-based localization training, we address cross-attention misalignment, achieving improved accuracy in binding multiple attributes to objects. Our approaches outperform existing compositional text-to-image diffusion models on the benchmark T2I CompBench, achieving better image quality, evaluated by humans, and emerging robustness under scaling number of binding pairs in the prompt.

Figures (8)

• Figure 1: Left: Original pipeline and generated image of Diffusion Model. Right: VSC individually "search" for visual information of each binding pair by generating images for each separate pair ("red car" and "yellow bicycle"). Finally, VSC uses them as references to generate the correct compositional image.
• Figure 2: The training pipeline of VSC. Given the input text prompt "A blue apple and a green backpack", VSC first generates images of each binding pair ("blue apple" and "green backpack"), then uses an image encoder to extract visual prototype features, which is later used to augment the text embeddings via a trainable MLP module. Finally, a frozen pre-trained diffusion model generates the image with the augmented text embeddings. Additionally, we compute the $L_{loc}$ loss between cross-attention maps and segmentation maps to enhance the precision of cross-attention maps.
• Figure 3: Generation with the increasing number of binding pairs. We show the generated images when the prompt includes more pairs. We observe that the generated image consistently reflects the additional composition.
• Figure 4: Qualitative result of compositionally. Compared to the baselines, our method can generate the image with better composition. Specifically, for the last row, Stable Diffusion 3.5 and VSC with 2.1 and 3.5 reflect the prompt better than others, such as silver refrigerator and wood chairs. For the second row, VSC with 2.1 and 3.5 generates the mirror; Stable Diffusion 3.5 fails to reflect the mirror.
• Figure 5: Scaling dataset. The accuracy of models on T2I-CompBench while training on different dataset sizes, highlighting the positive impact of dataset scaling on model performance.