CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models

TL;DR

CONFORM tackles semantic fidelity gaps in text-to-image diffusion by introducing a training-free, contrastive framework that operates on cross-attention maps during test-time. By formulating an InfoNCE loss over object-attribute attention pairs and incorporating maps from successive timesteps (and previous iterations), CONFORM steers latent updates to produce images that faithfully reflect prompts for both Stable Diffusion and Imagen. Extensive qualitative and quantitative evaluations against strong baselines, plus user studies, demonstrate improved object presence, attribute binding, and correct counts across diverse prompts. The approach is model-agnostic, lightweight to apply, and comes with open-source code to enable broader adoption and further research.

Abstract

Images produced by text-to-image diffusion models might not always faithfully represent the semantic intent of the provided text prompt, where the model might overlook or entirely fail to produce certain objects. Existing solutions often require customly tailored functions for each of these problems, leading to sub-optimal results, especially for complex prompts. Our work introduces a novel perspective by tackling this challenge in a contrastive context. Our approach intuitively promotes the segregation of objects in attention maps while also maintaining that pairs of related attributes are kept close to each other. We conduct extensive experiments across a wide variety of scenarios, each involving unique combinations of objects, attributes, and scenes. These experiments effectively showcase the versatility, efficiency, and flexibility of our method in working with both latent and pixel-based diffusion models, including Stable Diffusion and Imagen. Moreover, we publicly share our source code to facilitate further research.

Figures (29)

• Figure 1: Our training-free method combines a contrastive objective with test-time optimization, significantly improving how models such as Imagen and Stable Diffusion generate images with text prompts consisting of multiple concepts or subjects such as 'a bear and a horse'.
• Figure 2: Failure cases of Stable Diffusion stable-diffusion and Imagen imagen. Text-to-image diffusion models may not faithfully adhere to the subjects specified in the text prompt: a) missing objects (e.g., bear), b) misaligned attributes (e.g., the color yellow blends into the crown), and c) inaccurate object count (e.g., only one cat is generated instead of two). Our method steers the diffusion process towards more faithful images in both SD and Imagen.
• Figure 3: Attention scattering in backward process. In Stable Diffusion, the attention to attributes like green and yellow dissolves over backward timesteps (a). Our method effectively preserves these attention maps (b).
• Figure 4: An overview of CONFORM. Given a prompt (e.g., 'A green glasses and a yellow clock'), we extract the subject tokens green, glasses, yellow, and clock and their corresponding attention maps ($A^{\texttt{green}}, A^{\texttt{glasses}}, A^{\texttt{yellow}}, A^{\texttt{clock}}$) from timesteps $t$ and $t+1$. We employ our contrastive objective at each time step to repel negative pairs and attract positive pairs.
• Figure 5: Qualitative comparison of CONFORM on Stable Diffusion with other state-of-the-art methods. Our method generates more faithful images for the input text prompt on both simple and complex prompts.