Attention Calibration for Disentangled Text-to-Image Personalization

TL;DR

This work tackles the problem of learning and disentangling multiple novel concepts from a single reference image for personalized text-to-image generation. It introduces DisenDiff, an attention-calibration framework that jointly learns new modifier-class tokens, binds modifiers to their corresponding classes, and enforces separation and strengthening of cross-attention maps to achieve independent, coherent concept representations. By updating only the cross-attention keys/values and new embeddings within a Stable Diffusion backbone, it achieves high image fidelity while preserving editing flexibility, and demonstrates compatibility with inpainting and LoRA. Extensive experiments across ten datasets show superior image-alignment and competitive text-alignment, with ablations validating the contribution of each component and applications extending to three-concept scenarios and practical editing tasks.

Abstract

Recent thrilling progress in large-scale text-to-image (T2I) models has unlocked unprecedented synthesis quality of AI-generated content (AIGC) including image generation, 3D and video composition. Further, personalized techniques enable appealing customized production of a novel concept given only several images as reference. However, an intriguing problem persists: Is it possible to capture multiple, novel concepts from one single reference image? In this paper, we identify that existing approaches fail to preserve visual consistency with the reference image and eliminate cross-influence from concepts. To alleviate this, we propose an attention calibration mechanism to improve the concept-level understanding of the T2I model. Specifically, we first introduce new learnable modifiers bound with classes to capture attributes of multiple concepts. Then, the classes are separated and strengthened following the activation of the cross-attention operation, ensuring comprehensive and self-contained concepts. Additionally, we suppress the attention activation of different classes to mitigate mutual influence among concepts. Together, our proposed method, dubbed DisenDiff, can learn disentangled multiple concepts from one single image and produce novel customized images with learned concepts. We demonstrate that our method outperforms the current state of the art in both qualitative and quantitative evaluations. More importantly, our proposed techniques are compatible with LoRA and inpainting pipelines, enabling more interactive experiences.

Figures (13)

• Figure 1: Given one individual image from specific users, our proposed method is capable of producing customized images for each concept contained in the input image, e.g., given a single input image with a man and a woman, our method excels in achieving innovative renditions of both combined (left) and independent (right) concepts, without compromising the fidelity and identity preservation, and more importantly, manifesting satisfactory interactive generation conditioned by various text prompts. Note that we employ notation $V_i^*$ to denote the modifier of the $i$-th concept. Our code and data will be publicly available at: https://github.com/Monalissaa/DisenDiff.
• Figure 2: Failure case of Custom Diffusion kumari2023multi. In the third column, we show the example encompassing two failure settings: appearance inconsistency with the input image and ambiguous object not included in the target text. In the second column, we show the result from our method.
• Figure 3: Method overview. Our method applies constraints to the cross-attention maps of crucial tokens, ensuring the accurate representation of multiple concepts. We introduce new modifiers, denoted as $V_i^*$, along with the $i$-th class name, to represent the $i$-th personalized concept. Our attention calibration mechanism mainly includes three parts: the suppression technique performs self-sharpening and filters noisy small patches, the $\mathcal{L}_{\text{bind}}$ loss steers new modifiers towards the corresponding classes, and the $\mathcal{L}_{\text{s\&s}}$ loss guarantees the independence and completeness of the learned concepts.
• Figure 4: Comparison of generated attention maps and images. The first row displays the results of Custom Diffusion kumari2023multi, while the second row shows our results. During the training stage, when we obtain accurate attention maps for important tokens (left), it leads to the ideal output during the inference stage (right), maintaining high-concept similarity with the input image.
• Figure 5: Qualitative results of independent (left) and combined (right) concepts. The target prompt in each row represents a distinct context including learned concepts. Our method shows the highest visual similarity to the input image compared to Custom Diffusion and DreamBooth (especially in the first row, the results containing the specific toy) while preserving robust editability. Additionally, we show the ability to address the language drift issue and the disentanglement capability on the left of the second and last row, respectively.