Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Turn That Frown Upside Down: FaceID Customization via Cross-Training Data

Shuhe Wang, Xiaoya Li, Xiaofei Sun, Guoyin Wang, Tianwei Zhang, Jiwei Li, Eduard Hovy

TL;DR

This work tackles the limitation that FaceID customization models can only reproduce the exact input face by introducing CrossFaceID, a large-scale dataset of 40,000 text-image pairs across ~2,000 celebrities, paired with a cross-training strategy to learn controlled facial variations. By finetuning state-of-the-art FaceID customization models on CrossFaceID, the authors demonstrate that identity fidelity is preserved while customization capabilities—such as changing expressions, poses, or accessories—are significantly enhanced. Quantitative results on CrossFaceID-test and Unsplash-50, along with human evaluations, show improved prompt adherence and richer edits without sacrificing facial identity. The authors publicly release code, datasets, and trained models to accelerate progress in FaceID customization.

Abstract

Existing face identity (FaceID) customization methods perform well but are limited to generating identical faces as the input, while in real-world applications, users often desire images of the same person but with variations, such as different expressions (e.g., smiling, angry) or angles (e.g., side profile). This limitation arises from the lack of datasets with controlled input-output facial variations, restricting models' ability to learn effective modifications. To address this issue, we propose CrossFaceID, the first large-scale, high-quality, and publicly available dataset specifically designed to improve the facial modification capabilities of FaceID customization models. Specifically, CrossFaceID consists of 40,000 text-image pairs from approximately 2,000 persons, with each person represented by around 20 images showcasing diverse facial attributes such as poses, expressions, angles, and adornments. During the training stage, a specific face of a person is used as input, and the FaceID customization model is forced to generate another image of the same person but with altered facial features. This allows the FaceID customization model to acquire the ability to personalize and modify known facial features during the inference stage. Experiments show that models fine-tuned on the CrossFaceID dataset retain its performance in preserving FaceID fidelity while significantly improving its face customization capabilities. To facilitate further advancements in the FaceID customization field, our code, constructed datasets, and trained models are fully available to the public.

Turn That Frown Upside Down: FaceID Customization via Cross-Training Data

TL;DR

This work tackles the limitation that FaceID customization models can only reproduce the exact input face by introducing CrossFaceID, a large-scale dataset of 40,000 text-image pairs across ~2,000 celebrities, paired with a cross-training strategy to learn controlled facial variations. By finetuning state-of-the-art FaceID customization models on CrossFaceID, the authors demonstrate that identity fidelity is preserved while customization capabilities—such as changing expressions, poses, or accessories—are significantly enhanced. Quantitative results on CrossFaceID-test and Unsplash-50, along with human evaluations, show improved prompt adherence and richer edits without sacrificing facial identity. The authors publicly release code, datasets, and trained models to accelerate progress in FaceID customization.

Abstract

Existing face identity (FaceID) customization methods perform well but are limited to generating identical faces as the input, while in real-world applications, users often desire images of the same person but with variations, such as different expressions (e.g., smiling, angry) or angles (e.g., side profile). This limitation arises from the lack of datasets with controlled input-output facial variations, restricting models' ability to learn effective modifications. To address this issue, we propose CrossFaceID, the first large-scale, high-quality, and publicly available dataset specifically designed to improve the facial modification capabilities of FaceID customization models. Specifically, CrossFaceID consists of 40,000 text-image pairs from approximately 2,000 persons, with each person represented by around 20 images showcasing diverse facial attributes such as poses, expressions, angles, and adornments. During the training stage, a specific face of a person is used as input, and the FaceID customization model is forced to generate another image of the same person but with altered facial features. This allows the FaceID customization model to acquire the ability to personalize and modify known facial features during the inference stage. Experiments show that models fine-tuned on the CrossFaceID dataset retain its performance in preserving FaceID fidelity while significantly improving its face customization capabilities. To facilitate further advancements in the FaceID customization field, our code, constructed datasets, and trained models are fully available to the public.
Paper Structure (26 sections, 11 equations, 6 figures, 3 tables)

This paper contains 26 sections, 11 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-image Models
  4. ID Customization
  5. Dataset Construction: CrossFaceID
  6. Image Collection
  7. Image Filtering
  8. Image Annotations with GPT-4o
  9. CrossFaceID Based FaceID Customization
  10. Preliminaries
  11. Diffusion Models
  12. IP-Adapter
  13. InstantID
  14. FaceID Embedding.
  15. Spatial Facial Information Extraction.
  16. ...and 11 more sections

Figures (6)

  • Figure 1: An example illustrating the limitations of existing FaceID customization models in generating images of the same individual with variations. Here, we use the same image of a girl as the input, paired with two different text prompts. In Case 1, the desired output is an image of the person with smiling, and in Case 2, the goal is an image of the person facing right and wearing sunglasses. The results clearly show that the two leading FaceID customization models, IP-Adapter ye2023ip and InstantID wang2024instantid, perform well in preserving the exact same face as the input image but fail to customize the input face as specified—such as adding a smile for Case 1 or generating a right-facing person with sunglasses for Case 2. In contrast, the model trained on our proposed CrossFaceID dataset effectively addresses these shortcomings, successfully generating a smiling face for Case 1 and a right-facing person wearing sunglasses for Case 2.
  • Figure 2: Images sampled from our constructed CrossFaceID dataset are presented. From top to bottom, the display includes six persons, each occupying one row with six images showing different expressions and angles: (1) No Expression: a straight-faced person with no expression, (2) Smile: a person with a smiling face, (3) Sad: a person with a sad expression, (4) Rise: a person with his or her face rising up, (5) Side: a person with his or her face turned to one side, and (6) Wearing Glasses: a person wearing glasses.
  • Figure 3: The distribution of various facial features (e.g., expressions and angles) within our CrossFaceID dataset.
  • Figure 4: The results demonstrate the performance of FaceID customization models in maintaining FaceID fidelity. For models, "InstantID" refers to the official InstantID model, while "InstantID + CrossFaceID" represents the model further fine-tuned on our CrossFaceID dataset. "LAION" denotes the InstantID model pre-trained on our curated LAION dataset, and "LAION + CrossFaceID" refers to the model further trained on the CrossFaceID dataset. These results indicate that (1) for both the official InstantID model and the LAION-trained model, the ability to maintain FaceID fidelity remains consistent before and after fine-tuning on our CrossFaceID dataset, and (2) the model trained on our curated LAION dataset achieves comparable performance to the official InstantID model in preserving FaceID fidelity.
  • Figure 5: The results of the performance for FaceID customization models in customizing or editing FaceID. Here, "InstantID" represents the official InstantID model, while "InstantID + CrossFaceID" refers to the model fine-tuned on our CrossFaceID dataset. Similarly, "LAION" denotes the InstantID model pre-trained on our curated LAION dataset, and "LAION + CrossFaceID" refers to the model further fine-tuned on the CrossFaceID dataset. From these results, we can clearly observe an improvement in the models' ability to customize FaceID after being fine-tuned on our constructed CrossFaceID dataset.
  • ...and 1 more figures