Personalized Residuals for Concept-Driven Text-to-Image Generation

TL;DR

This work tackles open-domain personalization of text-to-image diffusion models by encoding a concept's identity through low-rank residuals applied to a small subset of weights, enabling rapid, reg-image-free customization. It introduces Localized Attention-Guided (LAG) sampling, which uses cross-attention maps to restrict the residual application to regions corresponding to the target concept, thereby preserving background and prior content. The approach uses approximately $1.2$M trainable parameters (about $0.1\%$ of the base model) and requires ~3 minutes on a single GPU, outperforming or matching baselines like Textual Inversion, ViCo, DreamBooth, and Custom Diffusion in both CLIP/DINO-based alignment and human preference metrics. The combination of low-rank residuals and on-the-fly localization enables efficient, flexible personalization across arbitrary concepts without regularization images, with practical implications for rapid, user-friendly creative workflows.

Abstract

We present personalized residuals and localized attention-guided sampling for efficient concept-driven generation using text-to-image diffusion models. Our method first represents concepts by freezing the weights of a pretrained text-conditioned diffusion model and learning low-rank residuals for a small subset of the model's layers. The residual-based approach then directly enables application of our proposed sampling technique, which applies the learned residuals only in areas where the concept is localized via cross-attention and applies the original diffusion weights in all other regions. Localized sampling therefore combines the learned identity of the concept with the existing generative prior of the underlying diffusion model. We show that personalized residuals effectively capture the identity of a concept in ~3 minutes on a single GPU without the use of regularization images and with fewer parameters than previous models, and localized sampling allows using the original model as strong prior for large parts of the image.

Figures (14)

• Figure 1: (Top) Given a set of reference images, we learn personalized residuals for a subset of a pretrained diffusion model's weights for efficient concept-driven text-to-image generation. (Bottom) The residuals can be combined with our proposed localized attention-guided (LAG) sampling, which leverages the cross-attention maps from the diffusion models to localize the application of the residuals and uses the original, unchanged, diffusion model for generating everything else.
• Figure 2: Overview of our proposed work. (1) Personalized residuals: We learn low-rank residuals for the output projection layer within each transformer block in the diffusion model. The residuals contain relatively few parameters, are fast to train, and do not require any regularization images during training. (2) Localized attention-guided sampling: We optionally apply the personalized residuals only in the areas that the cross-attention layers have localized the concept via predicted attention maps. Thus, we can combine the newly learned concept with the original generative prior of the base diffusion model within a single image.
• Figure 3: Qualitative comparison of our proposed approach with the baselines.
• Figure 4: Comparison of image generated with and without LAG sampling. We use the same starting noise map to generate corresponding pairs of images to directly visualize how LAG sampling affects the output image.
• Figure 5: AMT text alignment scores per prompt type.