A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization

TL;DR

This work addresses the challenge of preserving a subject's identity in diffusion-based personalization by introducing a data-centric regularization dataset generation strategy. By generating structured prompts and background-rich images via LLMs and coupling them with dropout and cropping, the method stabilizes fine-tuning without changing model architecture, achieving state-of-the-art identity preservation and text alignment on DreamBench. Experimental results reveal robust improvements across subjects and prompts, including fine details like logos, and demonstrate motion adaptation for living entities, though at the cost of increased training time and data generation. The approach is architecture-agnostic and broadly applicable to pre-trained diffusion models, offering a data-centric pathway for enhanced personalized generation.

Abstract

Large text-to-image models have revolutionized the ability to generate imagery using natural language. However, particularly unique or personal visual concepts, such as pets and furniture, will not be captured by the original model. This has led to interest in how to personalize a text-to-image model. Despite significant progress, this task remains a formidable challenge, particularly in preserving the subject's identity. Most researchers attempt to address this issue by modifying model architectures. These methods are capable of keeping the subject structure and color but fail to preserve identity details. Towards this issue, our approach takes a data-centric perspective. We introduce a novel regularization dataset generation strategy on both the text and image level. This strategy enables the model to preserve fine details of the desired subjects, such as text and logos. Our method is architecture-agnostic and can be flexibly applied on various text-to-image models. We show on established benchmarks that our data-centric approach forms the new state of the art in terms of identity preservation and text alignment.

Figures (16)

• Figure 1: Demonstration of our method applied to both SDXL and SD, compared with several prior works. Note that our results preserves very fine details of the subject without overfitting to the training images and losing text alignment.
• Figure 2: Overview. In contrast to prior approaches, we introduce specific prompts to the training examples and create a regularization dataset with a wider range of images guided by these prompts. To further boost diversity of regularization dataset, we generate additional prompts and images using the same prompt formats.
• Figure 3: Rephrasing prompts. Even when using rephrased prompts, our method maintains subject identity, a quality DreamBooth lacks. Notably, SDXL consistently generates images with minimal deviation from the originals.
• Figure 4: Influence of adaption to living entities. Without adaptation, the model might overlook motion in the prompts and focus solely on assembling objects within the images.
• Figure 5: Overfitting Prevention. Our regularization dataset effectively prevents the model from overfitting to the training images. Interestingly, when we employ a regularization dataset with a simplistic prompt "a [class noun]", the model exhibits improved text alignment during initial iterations, but experiences a decline in performance over time.