CoRe: Context-Regularized Text Embedding Learning for Text-to-Image Personalization

TL;DR

CoRe tackles the mismatch between identity preservation and text alignment in text-to-image personalization by regularizing the context tokens surrounding a new concept in the CLIP text encoder. By imposing context-embedding and context-attention constraints, and applying an embedding-rescaling scheme within a two-stage training framework, CoRe yields embeddings that generalize across prompts and improve alignment without requiring image generation during training. Empirical results show CoRe outperforms state-of-the-art baselines in both identity preservation and text alignment, with additional gains from test-time optimization. The approach provides a practical, generalizable path to high-quality, prompt-consistent personalization in diffusion-based image synthesis.

Abstract

Recent advances in text-to-image personalization have enabled high-quality and controllable image synthesis for user-provided concepts. However, existing methods still struggle to balance identity preservation with text alignment. Our approach is based on the fact that generating prompt-aligned images requires a precise semantic understanding of the prompt, which involves accurately processing the interactions between the new concept and its surrounding context tokens within the CLIP text encoder. To address this, we aim to embed the new concept properly into the input embedding space of the text encoder, allowing for seamless integration with existing tokens. We introduce Context Regularization (CoRe), which enhances the learning of the new concept's text embedding by regularizing its context tokens in the prompt. This is based on the insight that appropriate output vectors of the text encoder for the context tokens can only be achieved if the new concept's text embedding is correctly learned. CoRe can be applied to arbitrary prompts without requiring the generation of corresponding images, thus improving the generalization of the learned text embedding. Additionally, CoRe can serve as a test-time optimization technique to further enhance the generations for specific prompts. Comprehensive experiments demonstrate that our method outperforms several baseline methods in both identity preservation and text alignment. Code will be made publicly available.

Figures (16)

• Figure 1: CoRe enables text-aligned personalized generations, allowing for high visual variability of the user-provided concept.
• Figure 2: For the four similar prompts ("{} in the desert"), we show the cosine similarity between the output embeddings of each token (left), and the cross-attention map visualization of each token (right). Replacing "dog" with "puppy" or "cat" results in similar output embeddings and attention maps for other tokens. In contrast, using the overfitted $S_*$ by Textual Inversion significantly alters the output embeddings and attention maps for other tokens.
• Figure 3: Overview of the proposed CoRe. Our method enhances the text embedding learning for $S_*$ by regularizing its context tokens. Specifically, we randomly select a regularization prompt (e.g., "$S_*$ in the desert") and a reference prompt (e.g., "Dog in the desert") from the prompt set. During training, the proposed context embedding regularization and context attention regularization are applied together with the diffusion loss, which encourages the representations of the context tokens surrounding $S_*$ to align with those in the reference prompt. These regularization terms make the text embedding of $S_*$ more compatible with existing tokens.
• Figure 4: Qualitative comparison. We present personalization results of our method and four baseline methods, including Custom Diffusion kumari2022customdiffusion, NeTI alaluf2023neural, OFT qiu_oft, and AttnDreamBooth pang2024attndreambooth. Our method demonstrates superior performance in both text alignment and identity preservation compared to these baselines, especially for the prompts that require high visual variability of the concept.
• Figure 5: Face personalization results of our method and three baseline methods, including Cross Initialization (CI) pang2024cross, PhotoMaker (PM) li2023photomaker, and Face2Diffusion (FD) shiohara2024face2diffusion. Our method achieves more identity-preserved face generations compared to the baselines, especially when the input image is a side face.