GPTFace: Generative Pre-training of Facial-Linguistic Transformer by Span Masking and Weakly Correlated Text-image Data

TL;DR

GPTFace addresses the challenge of learning broad facial knowledge from large-scale, weakly correlated text-image data by proposing a generative pre-training framework that jointly models text and face images. It introduces span image-text masking (MILM) and image-text matching (ITM) within a shared Transformer backbone, coupled with a gradient-based controllable generation mechanism that uses ITM guidance to steer non-autoregressive generation. The approach achieves competitive performance on facial attribute classification and expression recognition, while enabling a range of face editing and image-to-text generation tasks such as text-guided editing, image tagging, and inpainting. By training on LAION-FACE with a VQGAN-based discrete image representation, GPTFace demonstrates rapid convergence and versatile applicability to both analytical and creative facial tasks, suggesting a scalable path for facial knowledge pre-training from web data. The combination of span masking, ITM supervision, and gradient-based control provides a practical framework for integrated face understanding and generation in real-world applications.

Abstract

Compared to the prosperity of pre-training models in natural image understanding, the research on large-scale pre-training models for facial knowledge learning is still limited. Current approaches mainly rely on manually assembled and annotated face datasets for training, but labeling such datasets is labor-intensive and the trained models have limited scalability beyond the training data. To address these limitations, we present a generative pre-training model for facial knowledge learning that leverages large-scale web-built data for training. We use texts and images containing human faces crawled from the internet and conduct pre-training on self-supervised tasks, including masked image/language modeling (MILM) and image-text matching (ITM). During the generation stage, we further utilize the image-text matching loss to pull the generation distribution towards the control signal for controllable image/text generation. Experimental results demonstrate that our model achieves comparable performance to state-of-the-art pre-training models for various facial downstream tasks, such as attribution classification and expression recognition. Furthermore, our approach is also applicable to a wide range of face editing tasks, including face attribute editing, expression manipulation, mask removal, and photo inpainting.

Figures (9)

• Figure 1: The LAION-FACE dataset zheng2022general, which is a subset of LAION-400M schuhmann2021laion, contains samples of facial image-text pairs that showcase the challenge of utilizing text information for human faces obtained from internet text-image pairs as the related text information is often uninformative.
• Figure 2: Overview of our approach. During pre-training, our model learns to generate text and images using shared parameters and masked image/language modeling tasks. The image-text matching objective helps to model the association between text and image, and guide the generator to achieve conditional image/text generation during inference.
• Figure 3: Visual examples of random masking and span masking on a text-image pair.
• Figure 4: Results of text-guided face editing. The masked region is highlighted in green, and the model uses the provided text to generate predictions for the masked areas.
• Figure 5: A comparison with StyleCLIP patashnik2021styleclip, FaRL zheng2022general, and Stable Diffusion rombach2021highresolution.