Representation Learning and Identity Adversarial Training for Facial Behavior Understanding

TL;DR

This work tackles facial action unit (AU) detection by addressing data diversity and identity leakage. It introduces Face9M for large-scale masked autoencoder pretraining (FMAE) and Identity Adversarial Training (IAT) to enforce identity-invariant AU features, revealing that a strong regularization is essential. Empirically, FMAE achieves state-of-the-art AU results on BP4D, BP4D+, and DISFA and strong FER performance, while FMAE-IAT further improves AU scores by mitigating identity shortcuts. The findings demonstrate the practical impact of large diverse datasets and adversarial identity regularization on fine-grained facial analysis, and the authors release code to enable broader adoption and further research.

Abstract

Facial Action Unit (AU) detection has gained significant attention as it enables the breakdown of complex facial expressions into individual muscle movements. In this paper, we revisit two fundamental factors in AU detection: diverse and large-scale data and subject identity regularization. Motivated by recent advances in foundation models, we highlight the importance of data and introduce Face9M, a diverse dataset comprising 9 million facial images from multiple public sources. Pretraining a masked autoencoder on Face9M yields strong performance in AU detection and facial expression tasks. More importantly, we emphasize that the Identity Adversarial Training (IAT) has not been well explored in AU tasks. To fill this gap, we first show that subject identity in AU datasets creates shortcut learning for the model and leads to sub-optimal solutions to AU predictions. Secondly, we demonstrate that strong IAT regularization is necessary to learn identity-invariant features. Finally, we elucidate the design space of IAT and empirically show that IAT circumvents the identity-based shortcut learning and results in a better solution. Our proposed methods, Facial Masked Autoencoder (FMAE) and IAT, are simple, generic and effective. Remarkably, the proposed FMAE-IAT approach achieves new state-of-the-art F1 scores on BP4D (67.1\%), BP4D+ (66.8\%), and DISFA (70.1\%) databases, significantly outperforming previous work. We release the code and model at https://github.com/forever208/FMAE-IAT.

Figures (5)

• Figure 1: Architecture of Identity Adversarial Training. The AU head and ID head both are a linear classifier predicting the AUs and identity, respectively. The backbone $G_{f}(\cdot; \theta_{f})$ is the encoder of the pretrained FMAE. During training, the AU head is optimized by $\frac{\partial L_{au}}{\partial \theta_{f}}$ and the ID head is optimized by $\frac{\partial L_{id}}{\partial \theta_{f}}$. The gradient reverse layer multiplies the gradient by a negative value $-\lambda$ to unlearn the features capable of recognizing identities. Finally, the parameters of the backbone are optimized by the two forces: $-\lambda \frac{\partial L_{id}}{\partial \theta_{f}}$ and $\frac{\partial L_{au}}{\partial \theta_{f}}$.
• Figure 2: F1 results of FMAE using different model sizes on 12 AUs of the BP4D. Models pretrained on Face9M are better than the ones pretrained on ImageNet-1k. MAE paper does not train ViT-small on ImageNet-1k, thus this entry is missing.
• Figure 3: Identity recognition accuracy (%) evaluated by linear probing on the BP4D dataset. IAT greatly reduces the identity-related features learned by the network backbone $G_{f}(\cdot; \theta_{f})$.
• Figure 4: t-SNE visualization of the backbone features on BP4D dataset regarding the identity labels, each color stands for a subject. Only 20 subjects are visualized for readability even though BP4D contains 41 subjects. FMAE features are more identity-clustered than FMAE-IAT features
• Figure 5: F1 dynamics of FMAE and FMAE-IAT on BP4D+ during training. Fold-2 of BP4D+ is used for visualization.