Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects

TL;DR

This work tackles subject mixing in diffusion-based text-to-image synthesis by introducing Self-Cross Diffusion Guidance, a training-free method that penalizes overlap between aggregated self-attention maps and cross-attention maps across multiple image patches. By aggregating self-attention over patches with strong cross-attention and enforcing cross-attention–self-attention disjointness, the method reduces cross-subject leakage while remaining compatible with both UNet and transformer-style diffusion backbones. The authors release the Similar Subjects Dataset (SSD) and employ GPT-4o-based evaluation (TIFA-GPT4o) to quantify existence, recognizability, and mixing, reporting significant improvements over baselines on several benchmarks. The approach achieves high fidelity to prompts, improves subject disentanglement, and also reduces subject neglect as a byproduct, suggesting broad applicability to faithful multi-subject synthesis in diffusion models.

Abstract

Diffusion models achieved unprecedented fidelity and diversity for synthesizing image, video, 3D assets, etc. However, subject mixing is an unresolved issue for diffusion-based image synthesis, particularly for synthesizing multiple similar-looking subjects. We propose Self-Cross Diffusion Guidance to penalize the overlap between cross-attention maps and the aggregated self-attention map. Compared to previous methods based on self-attention or cross-attention alone, our guidance is more effective in eliminating subject mixing. What's more, our guidance addresses subject mixing for all relevant patches beyond the most discriminant one, e.g., the beak of a bird. For each subject, we aggregate self-attention maps of patches with higher cross-attention values. Thus, the aggregated self-attention map forms a region that the whole subject attends to. Our training-free method boosts the performance of both Unet-based and Transformer-based diffusion models such as the Stable Diffusion series. We also release a similar subjects dataset (SSD), a challenging benchmark, and utilize GPT-4o for automatic and reliable evaluation. Extensive qualitative and quantitative results demonstrate the effectiveness of our self-cross diffusion guidance.

Figures (14)

• Figure 1: Results of Stable Diffusion latentdiffusionmodel and our method with Self-Cross guidance for the same prompt "a bear and an elephant". Images are generated from the same random seed. "cross" means cross-attention map and "self" means the aggregated self-attention map. The overlap between self-attention and cross-attention leads to subject mixing, while Self-Cross guidance reduces overlapping.
• Figure 2: Self-Cross diffusion guidance between the cross-attention of "elephant" and the self-attention of "bear"
• Figure 3: Qualitative comparisons of Self-Cross (ours) to SD1.4 latentdiffusionmodel, INITNO initno, CONFORM conform. For each prompt in the left column, we sample four seeds and show the results of different methods.
• Figure 4: Quantitative comparisons between original SD2.1 latentdiffusionmodel and our method.
• Figure 5: Quantitative comparisons between SD3-medium latentdiffusionmodel and our method.