General Facial Representation Learning in a Visual-Linguistic Manner

TL;DR

This work tackles the challenge of learning a universal facial representation transferable across multiple face analysis tasks. It introduces FaRL, a visual-linguistic pre-training framework combining image-text contrastive learning with masked image modeling, trained on LAION-FACE (20M face-image-text pairs). The approach yields superior transfer to face parsing, face alignment, and attribute prediction, including strong few-shot performance and state-of-the-art results on parsing and alignment. By using a frozen backbone and a 20M-scale, weakly supervised dataset, FaRL demonstrates data efficiency and practical potential for mobile and resource-constrained settings.

Abstract

How to learn a universal facial representation that boosts all face analysis tasks? This paper takes one step toward this goal. In this paper, we study the transfer performance of pre-trained models on face analysis tasks and introduce a framework, called FaRL, for general Facial Representation Learning in a visual-linguistic manner. On one hand, the framework involves a contrastive loss to learn high-level semantic meaning from image-text pairs. On the other hand, we propose exploring low-level information simultaneously to further enhance the face representation, by adding a masked image modeling. We perform pre-training on LAION-FACE, a dataset containing large amount of face image-text pairs, and evaluate the representation capability on multiple downstream tasks. We show that FaRL achieves better transfer performance compared with previous pre-trained models. We also verify its superiority in the low-data regime. More importantly, our model surpasses the state-of-the-art methods on face analysis tasks including face parsing and face alignment.

Figures (6)

• Figure 1: Image-text pairs randomly sampled from LAION-FACE. Web texts are not always accurate but often easier to acquire.
• Figure 2: Distribution of #faces in each image in LAION-FACE.
• Figure 3: Illustrating our pre-training framework. We integrate masked image modeling with image-text contrastive learning. The two $E_I$ in this figure stand for the same image encoder. After the pre-training, we use $E_I$ to boost downstream face tasks.
• Figure 4: Comparing FaRL with CLIP in text-driven face editing.
• Figure 5: Grad-CAM visualizations of $E_I$ given different text queries. Gradients are calculated in output of the first LayerNorm within the last Transformer block of $E_I$.