DOS: Directional Object Separation in Text Embeddings for Multi-Object Image Generation

TL;DR

This work tackles the failure of text-to-image models to faithfully render prompts containing multiple objects, caused by inter-object relationships. It introduces DOS, a lightweight method that computes directional separation vectors from pairwise object embeddings and injects them into three CLIP text embedding types (semantic token, EOT, and pooled) to promote object separation without modifying the generation process. By weighting separations with adaptive strengths derived from shape/texture and background cues, DOS significantly improves success rates and reduces object mixing across four benchmark scenarios, while maintaining fast inference comparable to baseline models. Human evaluations and cross-model validation corroborate the robustness and practical impact of the approach, including compatibility with SDXL and SD3.5 and potential synergy with latent-modification methods like Attend-and-Excite.

Abstract

Recent progress in text-to-image (T2I) generative models has led to significant improvements in generating high-quality images aligned with text prompts. However, these models still struggle with prompts involving multiple objects, often resulting in object neglect or object mixing. Through extensive studies, we identify four problematic scenarios, Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects, where inter-object relationships frequently lead to such failures. Motivated by two key observations about CLIP embeddings, we propose DOS (Directional Object Separation), a method that modifies three types of CLIP text embeddings before passing them into text-to-image models. Experimental results show that DOS consistently improves the success rate of multi-object image generation and reduces object mixing. In human evaluations, DOS significantly outperforms four competing methods, receiving 26.24%-43.04% more votes across four benchmarks. These results highlight DOS as a practical and effective solution for improving multi-object image generation.

Figures (18)

• Figure 1: Text-to-image generative models often struggle to generate images with multiple objects, especially in four scenarios where relationships between objects, referred to as inter-object relationships, frequently lead to object neglect or object mixing: Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects. Our method mitigates these failures by modifying text embeddings before feeding them into the text-to-image model.
• Figure 2: Analysis of four aspects of inter-object relationships in multi-object image generation. The top row shows quantitative comparisons between two contrasting conditions for each aspect, while the bottom row shows a qualitative example for each condition. A lower success rate indicates more frequent failures, such as object neglect or object mixing.
• Figure 3: Method Overview. DOS vectors are computed for the semantic token embeddings corresponding to object nouns, the EOT embedding, and the pooled embedding. These vectors are added to the corresponding CLIP text embeddings, namely $\bm{c}_{\mathrm{obj}}^{n}$ for $n=1,\dots,N$, $\bm{c}_{\mathrm{EOT}}$, and $\bm{c}_{\mathrm{pool}}$, before being passed into the text-to-image model. $N$ denotes the number of object nouns in the prompt.
• Figure 4: Qualitative comparison across four benchmarks: Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects. More results are shown in Figures \ref{['fig:qualitative_extension_1']}-\ref{['fig:qualitative_extension_3']} of Appendix \ref{['sec:appendix_additional_qualitative_results']}.
• Figure 5: Qualitative comparison on complex prompts. More results are shown in Figure \ref{['fig:general_prompt_extension']} of Appendix \ref{['sec:appendix_additional_qualitative_results']}.