Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis

TL;DR

Concept Conductor tackles multi-concept text-to-image synthesis by divorcing the denoising processes of individual concepts (multipath sampling) from a stable base model, then enforcing correct spatial layouts (layout alignment) and harmonious fusion of concepts (concept injection) via shape-aware masks and self-attention-guided masking. The approach preserves each concept’s visual fidelity while preventing attribute leakage and layout confusion, demonstrated on a new 30-concept dataset with strong gains over baseline methods in both text and image alignment. It leverages ED-LoRA for single-concept modeling, a gradient-guided reference-layout strategy, and attention-based feature fusion to generate coherent composite images across diverse and similar concepts. The work offers a training-free, scalable pathway for accurate multi-concept customization with practical implications for creative AI tooling, while also providing a dataset and metrics to benchmark multi-concept fidelity.

Abstract

The customization of text-to-image models has seen significant advancements, yet generating multiple personalized concepts remains a challenging task. Current methods struggle with attribute leakage and layout confusion when handling multiple concepts, leading to reduced concept fidelity and semantic consistency. In this work, we introduce a novel training-free framework, Concept Conductor, designed to ensure visual fidelity and correct layout in multi-concept customization. Concept Conductor isolates the sampling processes of multiple custom models to prevent attribute leakage between different concepts and corrects erroneous layouts through self-attention-based spatial guidance. Additionally, we present a concept injection technique that employs shape-aware masks to specify the generation area for each concept. This technique injects the structure and appearance of personalized concepts through feature fusion in the attention layers, ensuring harmony in the final image. Extensive qualitative and quantitative experiments demonstrate that Concept Conductor can consistently generate composite images with accurate layouts while preserving the visual details of each concept. Compared to existing baselines, Concept Conductor shows significant performance improvements. Our method supports the combination of any number of concepts and maintains high fidelity even when dealing with visually similar concepts. The code and models are available at https://github.com/Nihukat/Concept-Conductor.

Figures (17)

• Figure 1: Results from existing multi-concept customization methods (second row) and our method (top right). Our method aims to address attribute leakage and layout confusion (concept omission, subject redundancy, appearance truncation), producing visually faithful and text-aligned images.
• Figure 2: Overview of our proposed Concept Conductor. At each denoising step, the input latent vector $z_{t}$ is first corrected to $z_{t}\prime$ by the Layout Alignment module. $z_{t}\prime$ is then sent to the Concept Injection module for denoising, producing the next latent vector $z_{t-1}$. Both Layout Alignment and Concept Injection utilize the Multipath Sampling structure. After denoising, our method can generate images that align with the given text prompt and visual concepts.
• Figure 3: Illustration of multipath sampling. custom models $\epsilon_{\theta}^{V_{1}}$ and $\epsilon_{\theta}^{V_{2}}$ are created by adding ED-LoRA to the base model $\epsilon_{\theta}^{\text{base}}$. The base prompt and edited prompts are sent to the base model and custom models, respectively. Different models receive the same latent input $z_{t}$ and predict different noises. Self-attention features $F_{t}^{\text{base}}$, $F_{t}^{V_{1}}$, $F_{t}^{V_{2}}$, and the output feature maps of the attention layers $h_{t}^{\text{base}}$, $h_{t}^{V_{1}}$, $h_{t}^{V_{2}}$ are recorded during this process.
• Figure 4: Illustration of layout alignment. The self-attention feature $F_{t}^{\text{ref}}$ of the layout reference image is extracted through DDIM inversion, which is then used to compute the loss with $F_{t}^{\text{base}}$, $F_{t}^{V_{1}}$, and $F_{t}^{V_{2}}$, updating the input latent vector $z_{t}$. For simplicity, the conversion from pixel space to latent space is omitted.
• Figure 5: Illustration of concept injection, consisting of two parts: (1) Feature Fusion. The output feature maps of the attention layers from different models are multiplied by their corresponding masks and summed to obtain the fused feature map $h_{t}$, which is used to replace the original feature map $h_{t}^{\text{base}}$. (2) Mask Refinement. Segmentation maps are obtained by clustering on the self-attention, and the masks required for feature fusion are extracted from these maps.