Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment

TL;DR

Diffusion-based text-to-image models often misbind attributes to entities due to weak linguistic structure encoding. SynGen performs inference-time syntax-guided optimization by parsing prompts to identify entity-noun and modifier bindings and applying a cross-attention loss that aligns these bindings during the early diffusion steps, without retraining. Across ABC-6K, Attend-and-Excite, and the new DVMP dataset, human evaluations show substantial gains in proper binding and visual appeal, validating the approach. The work demonstrates the value of incorporating sentence structure into diffusion generation and suggests a general avenue for syntax-aware control in image synthesis.

Abstract

Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers in the prompt and visual binding of the corresponding elements in the generated image. As one notable example, a query like "a pink sunflower and a yellow flamingo" may incorrectly produce an image of a yellow sunflower and a pink flamingo. To remedy this issue, we propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses a novel loss function that encourages the cross-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining or fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods. This work highlights how making use of sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation.

Figures (24)

• Figure 1: Visual bindings of objects and their attributes may fail to match the linguistic bindings between entities and their modifiers. Our approach, SynGen, corrects these errors by matching the cross-attention maps of entities and their modifiers.
• Figure 2: The SynGen workflow and architecture. (a) The text prompt is analyzed to extract entity-nouns and their modifiers. (b) SynGen adds intermediates steps to the diffusion denoising process. In that step, we update the latent representation to minimize a loss over the cross attention maps of entity-nouns and their modifiers (Eq \ref{['eq:loss']}).
• Figure 2: Ablation of loss components. Values are percent preferred by human raters.
• Figure 3: Qualitative comparison for prompts from the Attend-and-Excite dataset. For every prompt, the same three seeds are used for all methods.
• Figure 4: Qualitative comparison for prompts from the DVMP dataset. For every prompt, the same three seeds are used for all methods.