Progressive Compositionality in Text-to-Image Generative Models

TL;DR

This work tackles the persistent challenge of compositionality in diffusion-based text-to-image generation by introducing ConPair, a 15k high-quality contrastive image-pair dataset curated with LLM-driven prompts and VQA-based alignment checks. It further proposes EvoGen, a three-stage curriculum for contrastive fine-tuning that progressively trains diffusion models on simple to complex compositions, guided by a projection head and an InfoNCE-style loss. Across extensive benchmarks and ablations, EvoGen achieves superior alignment and faithful object-attribute relationships, with the Stage-III curriculum driving gains on complex scenes. The combination of automatic, minimal-difference contrastive data and curriculum-based learning promises more reliable and interpretable compositional generation, with practical implications for robust T2I synthesis and downstream applications.

Abstract

Despite the impressive text-to-image (T2I) synthesis capabilities of diffusion models, they often struggle to understand compositional relationships between objects and attributes, especially in complex settings. Existing solutions have tackled these challenges by optimizing the cross-attention mechanism or learning from the caption pairs with minimal semantic changes. However, can we generate high-quality complex contrastive images that diffusion models can directly discriminate based on visual representations? In this work, we leverage large-language models (LLMs) to compose realistic, complex scenarios and harness Visual-Question Answering (VQA) systems alongside diffusion models to automatically curate a contrastive dataset, ConPair, consisting of 15k pairs of high-quality contrastive images. These pairs feature minimal visual discrepancies and cover a wide range of attribute categories, especially complex and natural scenarios. To learn effectively from these error cases, i.e., hard negative images, we propose EvoGen, a new multi-stage curriculum for contrastive learning of diffusion models. Through extensive experiments across a wide range of compositional scenarios, we showcase the effectiveness of our proposed framework on compositional T2I benchmarks.

Figures (15)

• Figure 1: Limited Compositionality Understanding in Diffusion Models. Existing SOTA models such as SDXL, DALL-E 3 often fail to correctly compose objects and attributes. The bottom are images generated by our EvoGen.
• Figure 2: EvoGen Framework. Data generation pipeline (left) and curriculum contrastive learning (right). Quality control of image generation (bottom): Given a prompt, SD3 generates multiple candidate images, which are evaluated by LLaVA. We select the best image by alignment and CLIPScore. If the alignment score is low, we prompt LLaVA to describe the image as a newly revised caption based on the generated image.
• Figure 3: Contrastive dataset examples. Each pair includes a positive image generated from the given prompt (left) and a negative image that is semantically inconsistent with the prompt (right), differing only minimally from the positive image.
• Figure 4: Average CLIP image-text similarities between the text prompts and the images generated by different models. The Full Prompt Similarity considers full-text prompt. Minimum Object represents the minimum of the similarities between the generated image and each of the two object prompts. An example of this benchmark is in \ref{['app:attn']}.
• Figure 5: Average CLIP similarity of image-text pairs in ConPair. Applying VQA checker consistently improves text-image alignment.