Dominating vs. Dominated: Generative Collapse in Diffusion Models

TL;DR

This work identifies Dominant-vs-Dominated (DvD) as a core failure mode in multi-concept text-to-image diffusion, where limited visual diversity in training data biases generation toward a dominant concept and suppresses others. It introduces DominanceBench, a dataset and protocol for systematic analysis of DvD, and provides a data-driven causal account showing that reduced visual diversity induces rigid priors that dominate compositional prompts. Through cross-attention and temporal analyses, the study reveals that dominance arises from early, layer-wise attention concentration on the dominant token and is distributed across multiple heads rather than localized to a few heads, complicating mitigation. The paper also proposes a real-time detection method using Focus Score and demonstrates that moderate architectural interventions, informed by these insights, are needed to address generative collapse without altering user prompts. Overall, the findings offer a principled pathway to more reliable, controllable diffusion-based generation in the presence of complex, multi-concept prompts.

Abstract

Text-to-image diffusion models have drawn significant attention for their ability to generate diverse and high-fidelity images. However, when generating from multi-concept prompts, one concept token often dominates the generation, suppressing the others-a phenomenon we term the Dominant-vs-Dominated (DvD) imbalance. To systematically analyze this imbalance, we introduce DominanceBench and examine its causes from both data and architectural perspectives. Through various experiments, we show that the limited instance diversity in training data exacerbates the inter-concept interference. Analysis of cross-attention dynamics further reveals that dominant tokens rapidly saturate attention, progressively suppressing others across diffusion timesteps. In addition, head ablation studies show that the DvD behavior arises from distributed attention mechanisms across multiple heads. Our findings provide key insights into generative collapse, advancing toward more reliable and controllable text-to-image generation.

Figures (16)

• Figure 1: Generation results for "Neuschwanstein Castle coaster" across five random seeds. Only one (SD 1.4) and two (SD 2.1) out of five seeds successfully generate both concepts.
• Figure 2: Training data examples from LAION. Neuschwanstein Castle exhibits minimal visual variation, while coasters appear in diverse forms and contexts.
• Figure 3: Comparison of mean DvD Scores between SD 1.4 and 2.1. Each box represents the distribution of mean DvD Scores across prompts (10 images per prompt). The red dashed line indicates the DvD Score threshold of 36.
• Figure 4: Generation results (left) and DvD scores (right, mean over 10 seeds) for two example prompts across training variants ($\mathcal{D}_1$, $\mathcal{D}_2$, $\mathcal{D}_4$, $\mathcal{D}_6$, $\mathcal{D}_8$, $\mathcal{D}_{10}$, baseline). The red dashed line in (b) and (d) indicates the DvD Score threshold of 36.
• Figure 5: Mean focus scores across UNet layers at the first denoising step for DominanceBench prompts and balanced prompts.