Isolated Diffusion: Optimizing Multi-Concept Text-to-Image Generation Training-Freely with Isolated Diffusion Guidance

TL;DR

This work tackles concept bleeding in multi-concept text-to-image diffusion by introducing Isolated Diffusion, a training-free approach that isolates the denoising of attachments and subjects via split prompts. It leverages GPT-4 to split prompts and uses pre-trained detectors (YOLO and SAM) to preserve layouts, applying per-concept denoising with explicit noise equations to avoid interference. Empirical results across qualitative, quantitative, and user studies show improved text-image consistency over baselines, with strong performance on SDXL and compatibility with prior Stable Diffusion models. The approach offers a practical, detector-assisted pathway to reliable multi-concept generation without additional training, enhancing applicability in real-world creative and editorial workflows.

Abstract

Large-scale text-to-image diffusion models have achieved great success in synthesizing high-quality and diverse images given target text prompts. Despite the revolutionary image generation ability, current state-of-the-art models still struggle to deal with multi-concept generation accurately in many cases. This phenomenon is known as ``concept bleeding" and displays as the unexpected overlapping or merging of various concepts. This paper presents a general approach for text-to-image diffusion models to address the mutual interference between different subjects and their attachments in complex scenes, pursuing better text-image consistency. The core idea is to isolate the synthesizing processes of different concepts. We propose to bind each attachment to corresponding subjects separately with split text prompts. Besides, we introduce a revision method to fix the concept bleeding problem in multi-subject synthesis. We first depend on pre-trained object detection and segmentation models to obtain the layouts of subjects. Then we isolate and resynthesize each subject individually with corresponding text prompts to avoid mutual interference. Overall, we achieve a training-free strategy, named Isolated Diffusion, to optimize multi-concept text-to-image synthesis. It is compatible with the latest Stable Diffusion XL (SDXL) and prior Stable Diffusion (SD) models. We compare our approach with alternative methods using a variety of multi-concept text prompts and demonstrate its effectiveness with clear advantages in text-image consistency and user study.

Figures (12)

• Figure 1: Multi-concept generated samples comparison. Our approach fixes the concept bleeding problem of SDXL.
• Figure 2: We isolate multi-subject generation by replacing the regions of other subjects in latents with random noises and denoise each subject with split text prompts individually. Our approach avoids mutual interference between various subjects and gets a more reasonable result than SDXL.
• Figure 3: Overview of Isolated Diffusion. We decompose complex text prompts into simpler forms with GPT4 and denoise each concept under split conditions to avoid mutual interference between various concepts for better text-image consistency.
• Figure 4: Qualitative comparison between our approach and baselines using text prompts of multiple attachments. Isolated Diffusion achieves the best text-image consistency among all the methods and maintains high fidelity similar to SDXL.
• Figure 5: Qualitative comparison between our approach and baselines using text prompts of multiple subjects. Isolated Diffusion revises the samples of SDXL to fix the problem of concept bleeding and achieves outstanding text-image consistency.