PIDiff: Image Customization for Personalized Identities with Diffusion Models

TL;DR

PIDiff tackles the challenge of personalized identity text-to-image generation by disentangling identity features from background and aligning with textual prompts. It introduces a Visual Guidance Module that leverages the StyleGAN W+ latent space and a Style Cross-Attention mechanism to fuse identity- and style-related cues into diffusion model generation, all within a targeted fine-tuning regime. A small identity-focused dataset and a pseudo-word identity token enable rapid learning of identity localization and style editing without semantic entanglement, achieving high identity fidelity and diversity. Experimental results show PIDiff outperforms state-of-the-art methods in identity preservation, prompt fidelity, and bias mitigation across identities, highlighting its practical potential for personalized image synthesis.

Abstract

Text-to-image generation for personalized identities aims at incorporating the specific identity into images using a text prompt and an identity image. Based on the powerful generative capabilities of DDPMs, many previous works adopt additional prompts, such as text embeddings and CLIP image embeddings, to represent the identity information, while they fail to disentangle the identity information and background information. As a result, the generated images not only lose key identity characteristics but also suffer from significantly reduced diversity. To address this issue, previous works have combined the W+ space from StyleGAN with diffusion models, leveraging this space to provide a more accurate and comprehensive representation of identity features through multi-level feature extraction. However, the entanglement of identity and background information in in-the-wild images during training prevents accurate identity localization, resulting in severe semantic interference between identity and background. In this paper, we propose a novel fine-tuning-based diffusion model for personalized identities text-to-image generation, named PIDiff, which leverages the W+ space and an identity-tailored fine-tuning strategy to avoid semantic entanglement and achieves accurate feature extraction and localization. Style editing can also be achieved by PIDiff through preserving the characteristics of identity features in the W+ space, which vary from coarse to fine. Through the combination of the proposed cross-attention block and parameter optimization strategy, PIDiff preserves the identity information and maintains the generation capability for in-the-wild images of the pre-trained model during inference. Our experimental results validate the effectiveness of our method in this task.

Figures (10)

• Figure 1: Overview of the proposed PIDiff. PIDiff consists of two modules: Visual Guidance Module(VGM) and the diffusion model. VGM processes the preprocessed image and outputs it to the diffusion model. The diffusion model is based on SDV1.5 and uses new transformer blocks to incorporate both text and visual prompts.
• Figure 2: Comparison of Image Generation Results and Attention Maps between Various Methods. The customized training strategy avoids semantic entanglement and effectively preserves key identity features.
• Figure 3: Illustration of Style Cross-Attention(SCA). SCA takes the output of the text cross-attention block as the query and uses the visual prompts as the keys and values. Only the projection matrices is trainable in SCAs.
• Figure 4: Illustration of inference process in diffusion models: Given a text prompt (e.g.," a photo of * reading in the library") and an identity image, PIDiff can generate images for the identity.
• Figure 5: Visualization of our proposed personalized identity customization dataset, we select samples from our dataset showcasing diverse identities, including variations in race, age, and gender.