Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generation

TL;DR

This work tackles object confusion in finetuning-free personalized image generation when multiple reference images are provided. It introduces a weighted-merge strategy that uses object-relevance signals derived from cross-attention to merge multiple reference features into appropriate objects, enabling training-free multi-object generation, and further strengthens performance with a training-based extension using a high-quality SA-1B–derived dataset and an object-quality score. The approach achieves state-of-the-art results on Concept101 and DreamBooth multi-object benchmarks and also boosts single-object personalization, while reducing training costs. Overall, the method provides a practical, scalable pathway to accurate multi-object personalization in diffusion-based image generation and comes with code for community use.

Abstract

Personalized text-to-image generation methods can generate customized images based on the reference images, which have garnered wide research interest. Recent methods propose a finetuning-free approach with a decoupled cross-attention mechanism to generate personalized images requiring no test-time finetuning. However, when multiple reference images are provided, the current decoupled cross-attention mechanism encounters the object confusion problem and fails to map each reference image to its corresponding object, thereby seriously limiting its scope of application. To address the object confusion problem, in this work we investigate the relevance of different positions of the latent image features to the target object in diffusion model, and accordingly propose a weighted-merge method to merge multiple reference image features into the corresponding objects. Next, we integrate this weighted-merge method into existing pre-trained models and continue to train the model on a multi-object dataset constructed from the open-sourced SA-1B dataset. To mitigate object confusion and reduce training costs, we propose an object quality score to estimate the image quality for the selection of high-quality training samples. Furthermore, our weighted-merge training framework can be employed on single-object generation when a single object has multiple reference images. The experiments verify that our method achieves superior performance to the state-of-the-arts on the Concept101 dataset and DreamBooth dataset of multi-object personalized image generation, and remarkably improves the performance on single-object personalized image generation. Our code is available at https://github.com/hqhQAQ/MIP-Adapter.

Figures (7)

• Figure 1: Left image demonstrates the object confusion problem in decoupled cross-attention mechanism, and right image presents the correct generation using our method.
• Figure 2: The reference image (IP) features, with the background masked, reduces generation quality in IP-Adapter.
• Figure 3: (A) demonstrates the calculation of $S_{\rm object\_relevance}$, which is used for selecting training data. The overall framework in (B) consists of a UNet model for noise prediction conditioned on the text prompt and multiple reference images. (C) presents the proposed weighted-merge method in each cross-attention layer of UNet from (B). $\tilde{\rm A}_{\rm img}^{i} = \frac{ {\rm A}_{\rm img}^{i} }{ \bar{\rm A}_{\rm img}^{i} }$, and $\tilde{f}(\mathbf{Z}_{\rm text}) = \frac{ f(\mathbf{Z}_{\rm text}) }{ \bar{f}(\mathbf{Z}_{\rm text}) }$.
• Figure 4: Qualitative comparisons of different methods on multi-object personalized image generation.
• Figure 5: Qualitative ablation experiment.