Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FaceSnap: Enhanced ID-fidelity Network for Tuning-free Portrait Customization

Benxiang Zhai, Yifang Xu, Guofeng Zhang, Yang Li, Sidan Du

TL;DR

FaceSnap addresses the challenge of high-fidelity, identity-preserving portrait generation in a zero-shot setting using a single reference image. It introduces a Facial Attribute Mixer to fuse CLIP-based appearance features with face ID embeddings, a Face Fidelity Reinforce Network that injects fused features along with 72-point landmarks, and a Landmark Predictor to preserve identity across poses, enabling one-shot customization without fine-tuning. Across experiments, it outperforms state-of-the-art methods in ID fidelity (FaceSim, CLIP-face) while maintaining competitive text alignment and generation quality, with practical deployment on GPUs with substantial VRAM. The work advances practical portrait customization by providing a generalizable, inference-efficient framework that can extend to different SD models and supports robust spatial control and identity preservation.

Abstract

Benefiting from the significant advancements in text-to-image diffusion models, research in personalized image generation, particularly customized portrait generation, has also made great strides recently. However, existing methods either require time-consuming fine-tuning and lack generalizability or fail to achieve high fidelity in facial details. To address these issues, we propose FaceSnap, a novel method based on Stable Diffusion (SD) that requires only a single reference image and produces extremely consistent results in a single inference stage. This method is plug-and-play and can be easily extended to different SD models. Specifically, we design a new Facial Attribute Mixer that can extract comprehensive fused information from both low-level specific features and high-level abstract features, providing better guidance for image generation. We also introduce a Landmark Predictor that maintains reference identity across landmarks with different poses, providing diverse yet detailed spatial control conditions for image generation. Then we use an ID-preserving module to inject these into the UNet. Experimental results demonstrate that our approach performs remarkably in personalized and customized portrait generation, surpassing other state-of-the-art methods in this domain.

FaceSnap: Enhanced ID-fidelity Network for Tuning-free Portrait Customization

TL;DR

FaceSnap addresses the challenge of high-fidelity, identity-preserving portrait generation in a zero-shot setting using a single reference image. It introduces a Facial Attribute Mixer to fuse CLIP-based appearance features with face ID embeddings, a Face Fidelity Reinforce Network that injects fused features along with 72-point landmarks, and a Landmark Predictor to preserve identity across poses, enabling one-shot customization without fine-tuning. Across experiments, it outperforms state-of-the-art methods in ID fidelity (FaceSim, CLIP-face) while maintaining competitive text alignment and generation quality, with practical deployment on GPUs with substantial VRAM. The work advances practical portrait customization by providing a generalizable, inference-efficient framework that can extend to different SD models and supports robust spatial control and identity preservation.

Abstract

Benefiting from the significant advancements in text-to-image diffusion models, research in personalized image generation, particularly customized portrait generation, has also made great strides recently. However, existing methods either require time-consuming fine-tuning and lack generalizability or fail to achieve high fidelity in facial details. To address these issues, we propose FaceSnap, a novel method based on Stable Diffusion (SD) that requires only a single reference image and produces extremely consistent results in a single inference stage. This method is plug-and-play and can be easily extended to different SD models. Specifically, we design a new Facial Attribute Mixer that can extract comprehensive fused information from both low-level specific features and high-level abstract features, providing better guidance for image generation. We also introduce a Landmark Predictor that maintains reference identity across landmarks with different poses, providing diverse yet detailed spatial control conditions for image generation. Then we use an ID-preserving module to inject these into the UNet. Experimental results demonstrate that our approach performs remarkably in personalized and customized portrait generation, surpassing other state-of-the-art methods in this domain.
Paper Structure (20 sections, 4 equations, 5 figures, 3 tables)

This paper contains 20 sections, 4 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. FaceSnap
  4. Overview
  5. Facial Attribute Mixer
  6. Face Fidelity Reinforce Network
  7. Training and Inference
  8. Experiments
  9. Experimental Settings
  10. Implementation details
  11. Evaluation metrics
  12. Comparisons
  13. Qualitative results
  14. Quantitative results
  15. Deployment feasibility
  16. ...and 5 more sections

Figures (5)

  • Figure 1: The overall framework of our proposed FaceSnap. The framework mainly consists of three modules: Facial Attribute Mixer, Face Fidelity Reinforce Network, and Landmark Predictor. We utilize pre-trained models to extract CLIP image features and face ID embeddings. The Facial Attribute Mixer then combines detailed facial information and facial structural information to extract comprehensive fused features. Facial landmarks serve as spatial control conditions, and the Face Fidelity Reinforce Network is introduced to encode both the spatial control conditions and the fused facial features. During inference, to maintain the fidelity of the generated face across different poses, we design the Landmark Predictor to generate landmarks retaining the facial structure of the reference image.
  • Figure 2: Qualitative comparison samples. We compare our FaceSnap with ConsistentIDConsistentID-2024, PhotomakerPhotoMaker-2023, IP-AdapterIP-Adapter-2023, PuLIDPulID-2024 and InstantIDInstantID-2024 using five distinct identities with corresponding prompts. It can be observed that our method has the ability to generate higher-quality and fidelity images.
  • Figure 3: User preferences on realistic level, ID fidelity for different methods
  • Figure 4: Impact of Facial Attribute Mixer. Compared with "concat with projection", using the Facial Attribute Mixer to obtain facial features effectively captures the ID details from Face ID embeddings and CLIP image features, thereby enhancing ID fidelity.
  • Figure 5: Effect of FFRNet and Landmark Predictor. FFRNet and landmarks used as spatial control conditions help with ID fidelity. When the face shapes of the reference and driving images differ significantly, the Landmark Predictor generates landmarks that reflect the face shape of the reference image, thereby improving ID fidelity.