Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

TL;DR

The paper tackles FER in the wild by combining multi-modal data (video, audio, text) with self-supervised learning in a multi-task framework. It introduces ConCluGen, which jointly optimizes multi-modal contrastive learning, online clustering, and data reconstruction to produce semantically meaningful, cross-modal representations without extensive labels. Through experiments on CAER, MELD, and MOSEI, ConCluGen achieves superior or competitive performance against state-of-the-art self-supervised and supervised baselines, with text modality providing particularly strong signals. The work demonstrates that multi-task, multi-modal SSL can reduce annotation needs while delivering robust FER representations, and it releases pre-trained models and code for broader use.

Abstract

Human communication is multi-modal; e.g., face-to-face interaction involves auditory signals (speech) and visual signals (face movements and hand gestures). Hence, it is essential to exploit multiple modalities when designing machine learning-based facial expression recognition systems. In addition, given the ever-growing quantities of video data that capture human facial expressions, such systems should utilize raw unlabeled videos without requiring expensive annotations. Therefore, in this work, we employ a multitask multi-modal self-supervised learning method for facial expression recognition from in-the-wild video data. Our model combines three self-supervised objective functions: First, a multi-modal contrastive loss, that pulls diverse data modalities of the same video together in the representation space. Second, a multi-modal clustering loss that preserves the semantic structure of input data in the representation space. Finally, a multi-modal data reconstruction loss. We conduct a comprehensive study on this multimodal multi-task self-supervised learning method on three facial expression recognition benchmarks. To that end, we examine the performance of learning through different combinations of self-supervised tasks on the facial expression recognition downstream task. Our model ConCluGen outperforms several multi-modal self-supervised and fully supervised baselines on the CMU-MOSEI dataset. Our results generally show that multi-modal self-supervision tasks offer large performance gains for challenging tasks such as facial expression recognition, while also reducing the amount of manual annotations required. We release our pre-trained models as well as source code publicly

Figures (3)

• Figure 1: Overview of the architecture of the multi-task multi-modal self-supervised method. The backbone feature extractors process the input modalities in blocks, which we average over the time domain. The resulting temporal features are stored on disk, i.e. the backbones are fully fixed. The 3-layer produces the representations we want to use in downstream training. The three losses are: (a) A reconstruction loss which reconstructs the features of each modality individually. (b) The multi-modal contrastive loss encourages the representations from the projection head for modalities belonging to the same input to be represented closer to each other. (c) The multi-modal clustering loss which drives the modalities of a sample towards the centroids computed by k-means clustering. The latter uses the mean of the modalities to compute these centroids.
• Figure 2: Confusion metrics for MELD dataset over different self-supervised models.
• Figure 3: Confusion metrics for CAER dataset over different models.