PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium

TL;DR

The paper addresses the fidelity-editability trade-off in one-shot face personalization with diffusion models. It introduces PersonaMagic, a stage-regulated conditioning framework that learns dynamic embeddings over a middle timestep window while static stages use supercategory embeddings, with a three-stage partition guided by IoU with facial masks. A Tandem Equilibrium strategy regularizes self-attention in the text encoder to balance the new concept with surrounding tokens, achieving semantically complete and identity-preserving results while keeping the diffusion model frozen. Extensive experiments show superior performance on one-shot and few-shot benchmarks, generalization to non-facial domains, and plug-in compatibility with pretrained personalization models like Photomaker and IP-Adapter.

Abstract

Personalized image generation has made significant strides in adapting content to novel concepts. However, a persistent challenge remains: balancing the accurate reconstruction of unseen concepts with the need for editability according to the prompt, especially when dealing with the complex nuances of facial features. In this study, we delve into the temporal dynamics of the text-to-image conditioning process, emphasizing the crucial role of stage partitioning in introducing new concepts. We present PersonaMagic, a stage-regulated generative technique designed for high-fidelity face customization. Using a simple MLP network, our method learns a series of embeddings within a specific timestep interval to capture face concepts. Additionally, we develop a Tandem Equilibrium mechanism that adjusts self-attention responses in the text encoder, balancing text description and identity preservation, improving both areas. Extensive experiments confirm the superiority of PersonaMagic over state-of-the-art methods in both qualitative and quantitative evaluations. Moreover, its robustness and flexibility are validated in non-facial domains, and it can also serve as a valuable plug-in for enhancing the performance of pretrained personalization models.

Figures (18)

• Figure 1: PersonaMagic seamlessly generates images of new roles, styles, or scenes based on a user-provided portrait. By learning stage-regulated embeddings through a Tandem Equilibrium strategy, our method accurately captures and represents unseen concepts, faithfully creating personas aligned with the provided prompts while minimizing identity distortion.
• Figure 2: Cross-attention maps of $S*$ at each timestep. We calculate IoU with facial mask for stage partition.
• Figure 3: Overview of our pipeline. Given a single image, we learn a series of embeddings during dynamic stage to capture identity information effectively, while employing fixed embeddings in the static stages. The proposed TE strategy is applied in the text encoder, ensuring further alignment of personalized results with textual descriptions.
• Figure 4: The overlooked semantic yield suboptimal attention map. Attention weights are annotated in the lower left corner of cross-attention maps.
• Figure 5: Illustration of the proposed Tandem Equilibrium.