PaCo-FR: Patch-Pixel Aligned End-to-End Codebook Learning for Facial Representation Pre-training

TL;DR

PaCo-FR is introduced, an unsupervised framework that combines masked image modeling with patch-pixel alignment that offers a scalable, efficient solution that reduces reliance on expensive annotated datasets, driving more effective facial analysis systems.

Abstract

Facial representation pre-training is crucial for tasks like facial recognition, expression analysis, and virtual reality. However, existing methods face three key challenges: (1) failing to capture distinct facial features and fine-grained semantics, (2) ignoring the spatial structure inherent to facial anatomy, and (3) inefficiently utilizing limited labeled data. To overcome these, we introduce PaCo-FR, an unsupervised framework that combines masked image modeling with patch-pixel alignment. Our approach integrates three innovative components: (1) a structured masking strategy that preserves spatial coherence by aligning with semantically meaningful facial regions, (2) a novel patch-based codebook that enhances feature discrimination with multiple candidate tokens, and (3) spatial consistency constraints that preserve geometric relationships between facial components. PaCo-FR achieves state-of-the-art performance across several facial analysis tasks with just 2 million unlabeled images for pre-training. Our method demonstrates significant improvements, particularly in scenarios with varying poses, occlusions, and lighting conditions. We believe this work advances facial representation learning and offers a scalable, efficient solution that reduces reliance on expensive annotated datasets, driving more effective facial analysis systems.

Figures (6)

• Figure 1: (a) The pipeline of FaRL zheng2022general, (b) The pipeline of MCF wang2023toward, and (c) our proposed PaCo-FR. PaCo-FR is specifically designed for facial characteristics and optimize it as a learning objective for fatial representation pre-training.
• Figure 2: Two key phenomena observed in facial image analysis. (a) After face alignment, various facial elements can be matched to corresponding positions, enhancing consistency. (b) Facial features, such as eyes, are clustered into subcategories based on attributes like makeup and state, enabling finer classification.
• Figure 3: The framework of PaCo-FR incorporates an incubation stage: During the initial epoch of training, we supervise the predictions of the Belief Predictor, encouraging it to learn the mapping relationship from pixel space to codebook space. A part of patches($t_{*}$) are randomly selected from images, and based on the original pixel values of these patches, suitable codebook tokens($t_{*}^*$) are predicted and selected. The image $\hat{I}$, with some patches replaced, undergoes encoder and decoder processes to predict the original image $I$.
• Figure 4: Cropped facial alignment results from the LAION-FACE dataset (top) and examples of noisy data present in the training set (bottom).
• Figure 5: The visualizations depict the impact of codebook size and different configurations on the generative capabilities of the model in the proposed method.