Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis

Zichuan Liu, Liming Jiang, Qing Yan, Yumin Jia, Hao Kang, Xin Lu

TL;DR

This work introduces FaceCLIP, a multi-modal encoder that learns a joint identity-text representation to condition ID-preserving portrait synthesis, avoiding adapters and enabling full diffusion-model adaptation. By aligning the joint embedding with face, text, and image spaces and fusing identity and semantic cues via a Fusion Module, FaceCLIP delivers stronger identity preservation and semantic alignment when integrated with Stable Diffusion XL as FaceCLIP-SDXL. The approach demonstrates state-of-the-art results on identity similarity and perceptual quality across internal and public portrait datasets, supported by zero-shot text alignment, identity clustering analyses, and user studies. The method offers a scalable, end-to-end framework for controllable, high-fidelity portrait synthesis with robust identity fidelity and semantic controllability.

Abstract

We propose a novel framework for ID-preserving generation using a multi-modal encoding strategy rather than injecting identity features via adapters into pre-trained models. Our method treats identity and text as a unified conditioning input. To achieve this, we introduce FaceCLIP, a multi-modal encoder that learns a joint embedding space for both identity and textual semantics. Given a reference face and a text prompt, FaceCLIP produces a unified representation that encodes both identity and text, which conditions a base diffusion model to generate images that are identity-consistent and text-aligned. We also present a multi-modal alignment algorithm to train FaceCLIP, using a loss that aligns its joint representation with face, text, and image embedding spaces. We then build FaceCLIP-SDXL, an ID-preserving image synthesis pipeline by integrating FaceCLIP with Stable Diffusion XL (SDXL). Compared to prior methods, FaceCLIP-SDXL enables photorealistic portrait generation with better identity preservation and textual relevance. Extensive experiments demonstrate its quantitative and qualitative superiority.

Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis

TL;DR

This work introduces FaceCLIP, a multi-modal encoder that learns a joint identity-text representation to condition ID-preserving portrait synthesis, avoiding adapters and enabling full diffusion-model adaptation. By aligning the joint embedding with face, text, and image spaces and fusing identity and semantic cues via a Fusion Module, FaceCLIP delivers stronger identity preservation and semantic alignment when integrated with Stable Diffusion XL as FaceCLIP-SDXL. The approach demonstrates state-of-the-art results on identity similarity and perceptual quality across internal and public portrait datasets, supported by zero-shot text alignment, identity clustering analyses, and user studies. The method offers a scalable, end-to-end framework for controllable, high-fidelity portrait synthesis with robust identity fidelity and semantic controllability.

Abstract

We propose a novel framework for ID-preserving generation using a multi-modal encoding strategy rather than injecting identity features via adapters into pre-trained models. Our method treats identity and text as a unified conditioning input. To achieve this, we introduce FaceCLIP, a multi-modal encoder that learns a joint embedding space for both identity and textual semantics. Given a reference face and a text prompt, FaceCLIP produces a unified representation that encodes both identity and text, which conditions a base diffusion model to generate images that are identity-consistent and text-aligned. We also present a multi-modal alignment algorithm to train FaceCLIP, using a loss that aligns its joint representation with face, text, and image embedding spaces. We then build FaceCLIP-SDXL, an ID-preserving image synthesis pipeline by integrating FaceCLIP with Stable Diffusion XL (SDXL). Compared to prior methods, FaceCLIP-SDXL enables photorealistic portrait generation with better identity preservation and textual relevance. Extensive experiments demonstrate its quantitative and qualitative superiority.

Paper Structure

This paper contains 19 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal Alignment and Fusion
  4. Customized Image Synthesis
  5. Methodology
  6. Problem Formulation
  7. Learning Joint Identity-text Encoding
  8. Pre-training Workflow
  9. Fusion Module
  10. Regularizing Pre-training with Image-text Data
  11. Training Diffusion Model with Joint Identity-text Priors
  12. Experiments
  13. Setting
  14. Text Alignment Verification
  15. Identity Alignment Verification
  16. ...and 4 more sections

Figures (3)

  • Figure 1: (a) FaceCLIP architecture and pre-training workflow.The modules labeled in red are freezed and the modules labeled in blue are unfreezed during pre-training; (b) Architecture of Feature Fusion Block in Fusion Module; (c) Detailed computation graph of Dual Cross-Attention.
  • Figure 2: Visualization of subspaces learned by FaceCLIP-L-14 and FaceCLIP-bigG-14. We visualize 500 aligned face images across 20 identities. (a) and (b) depict subspaces learned via diffusion loss without pre-training, whereas (c) and (d) illustrate subspaces from pre-trained FaceCLIP-L-14 and FaceCLIP-bigG-14, respectively.
  • Figure 3: Images produced by FaceCLIP-SDXL and PuLID-SDXL: The first row are the reference images and corresponding text prompts; The second row are the images generated by PuLID-SDXL; The third row are the images generated by FaceCLIP-SDXL.