SVFAP: Self-supervised Video Facial Affect Perceiver

TL;DR

Comprehensive results demonstrate that SVFAP can learn powerful affect-related representations via large-scale self-supervised pre-training and it significantly outperforms previous state-of-the-art methods on all datasets.

Abstract

Video-based facial affect analysis has recently attracted increasing attention owing to its critical role in human-computer interaction. Previous studies mainly focus on developing various deep learning architectures and training them in a fully supervised manner. Although significant progress has been achieved by these supervised methods, the longstanding lack of large-scale high-quality labeled data severely hinders their further improvements. Motivated by the recent success of self-supervised learning in computer vision, this paper introduces a self-supervised approach, termed Self-supervised Video Facial Affect Perceiver (SVFAP), to address the dilemma faced by supervised methods. Specifically, SVFAP leverages masked facial video autoencoding to perform self-supervised pre-training on massive unlabeled facial videos. Considering that large spatiotemporal redundancy exists in facial videos, we propose a novel temporal pyramid and spatial bottleneck Transformer as the encoder of SVFAP, which not only largely reduces computational costs but also achieves excellent performance. To verify the effectiveness of our method, we conduct experiments on nine datasets spanning three downstream tasks, including dynamic facial expression recognition, dimensional emotion recognition, and personality recognition. Comprehensive results demonstrate that SVFAP can learn powerful affect-related representations via large-scale self-supervised pre-training and it significantly outperforms previous state-of-the-art methods on all datasets. Code is available at https://github.com/sunlicai/SVFAP.

Figures (7)

• Figure 1: An overview of the proposed method (i.e., SVFAP). It consists of two stages, including self-supervised pre-training (a) and downstream fine-tuning (b). During pre-training, SVFAP utilizes masked facial video autoencoding as the training objective. Following previous studies he2022maskedtong2022videomaefeichtenhofer2022masked, it adopts an asymmetric encoder-decoder architecture and a high masking ratio (e.g., 90%) to enable fast pre-training on large-scale unlabeled facial video data. After pre-training, the lightweight decoder is discarded and the high-capacity encoder is fine-tuned in downstream tasks.
• Figure 2: The encoder in SVFAP, i.e., TPSBT. We perform summation-based multi-scale fusion for features from three stages during pre-training and empirically do not use it in fine-tuning. The temporal length in each stage $T_i = \frac{T}{2k^{i-1}}$ ($k$ is the downsampling rate and empirically $k=2$, $i \in \{1,2,3\}$). The spatial size $S = \frac{H}{16} \cdot \frac{W}{16} \cdot (1-\rho)$ ($\rho$ is the masking ratio and $\rho=90\%$) during pre-training and $S=\frac{H}{16} \cdot \frac{W}{16}$ during fine-tuning.
• Figure 3: The illustration of the face patch location for VoxCeleb2 videos.
• Figure 4: Ablation study of training from scratch.
• Figure 5: Ablation study of training schedule.