A Survey on Personalized Content Synthesis with Diffusion Models

TL;DR

This survey maps the rapid growth of Personalized Content Synthesis (PCS) in diffusion models, organizing methods into test-time fine-tuning (TTF) and pre-trained adaptation (PTA) while detailing techniques such as attention manipulation, masking, data augmentation, and regularization. It surveys PCS across object, style, and face personalization and extends to video and 3D domains, highlighting current limitations like overfitting and the fidelity-text alignment trade-off. The authors propose a unified evaluation landscape, including a new Persona benchmark, and discuss challenges around standardization, multimodal frameworks, and interactive workflows. Collectively, the work provides a comprehensive roadmap for advancing PCS with practical guidance for researchers and practitioners.

Abstract

Recent advancements in diffusion models have significantly impacted content creation, leading to the emergence of Personalized Content Synthesis (PCS). By utilizing a small set of user-provided examples featuring the same subject, PCS aims to tailor this subject to specific user-defined prompts. Over the past two years, more than 150 methods have been introduced in this area. However, existing surveys primarily focus on text-to-image generation, with few providing up-to-date summaries on PCS. This paper provides a comprehensive survey of PCS, introducing the general frameworks of PCS research, which can be categorized into test-time fine-tuning (TTF) and pre-trained adaptation (PTA) approaches. We analyze the strengths, limitations, and key techniques of these methodologies. Additionally, we explore specialized tasks within the field, such as object, face, and style personalization, while highlighting their unique challenges and innovations. Despite the promising progress, we also discuss ongoing challenges, including overfitting and the trade-off between subject fidelity and text alignment. Through this detailed overview and analysis, we propose future directions to further the development of PCS.

Figures (15)

• Figure 1: Given a few reference images of a subject (e.g., a cat 2ruiz2023dreambooth or face 54xiao2023fastcomposer), PCS aims to generate new renditions of the subject that align with user-defined textual prompts. The task requires preserving the subject’s identity while adapting to diverse contexts. The examples are generated by this survey using DreamBooth 2ruiz2023dreambooth and InstantID 37wang2024instantid.
• Figure 2: A chronological overview of classical PCS methods as surveyed, illustrating the evolution of techniques through months. The number of related works has rapidly increased over the past two years. We divide PCS methods with 3 different criteria: training strategy, personalization scope, and technique.
• Figure 3: The trade-off between text alignment and visual fidelity in personalized image synthesis, illustrated through DreamBooth-generated 2ruiz2023dreambooth examples of a customized cat wearing sunglasses. Overfitting occurs when the model focuses solely on reconstructing the cat, disregarding the sunglasses context. Underfitting, on the other hand, reflects the model's attempt to satisfy the text prompt but fails to accurately represent the personalized cat. Collapse signifies a failure to meet both criteria.
• Figure 4: Illustration of the TTF framework for the test-time fine-tuning process and generation phase. During the inference phase, the model fine-tunes its parameters by reconstructing the reference images for each SoI group. The unique modifier V* is employed to represent the SoI and used to formulate new inference prompts for generating personalized images.
• Figure 5: Illustration of the PTA method for personalized image synthesis. This framework utilizes a large-scale dataset to train a unified model that can process diverse personalization requests. The diffusion model is adapted to process hybrid inputs derived from both visual and textual features. Additionally, the concatenation of image and text features can be implemented in various ways, such as placeholder-based and reference-conditioned.