Multi-modal Transfer Learning for Dynamic Facial Emotion Recognition in the Wild

TL;DR

The paper tackles dynamic facial emotion recognition in unconstrained settings by leveraging multimodal transfer learning across visual, audio, and pose modalities. It presents a modular pipeline with pretrained ResNet-18, OpenPose, and Wave2Vec2 feature extractors, whose outputs are integrated through a transformer and a decision network, with a focus on late fusion for multimodal signals. Empirical results show that while end-to-end multimodal fusion inside the transformer can degrade performance, a decision-level fusion approach yields modest but meaningful gains, achieving 72.97% WAR and surpassing the DFEW baseline while approaching state-of-the-art. The findings highlight the importance of cross-modal alignment and fusion strategy in FER in-the-wild, offering practical guidance for designing robust multimodal emotion recognition systems.

Abstract

Facial expression recognition (FER) is a subset of computer vision with important applications for human-computer-interaction, healthcare, and customer service. FER represents a challenging problem-space because accurate classification requires a model to differentiate between subtle changes in facial features. In this paper, we examine the use of multi-modal transfer learning to improve performance on a challenging video-based FER dataset, Dynamic Facial Expression in-the-Wild (DFEW). Using a combination of pretrained ResNets, OpenPose, and OmniVec networks, we explore the impact of cross-temporal, multi-modal features on classification accuracy. Ultimately, we find that these finely-tuned multi-modal feature generators modestly improve accuracy of our transformer-based classification model.

Figures (6)

• Figure 1: Original Multi-modal FER model architecture. We combine three pretrained models to extract facial features, body-pose data, and audio-sentiment from the processed and raw clips. Then, we leverage a multilayer transformer encoder to learn spatiotemporal dependencies and relationships to improve FER classification accuracy. Ultimately, we opted to connect the Mel Spectrogram and OpenPose features directly to the FC layers after poor initial results.
• Figure 2: Model performance of the emotion classifier trained using Mean Squared Error (MSE) loss. Plots (a) and (c) show results from the first 10 epochs without regularization, while plots (b) and (d) represent performance with a regularization term (weight_decay) set to 0.0001.
• Figure 3: Model performance of the emotion classifier trained with KL Divergence (KL) loss is illustrated. Plots (a) and (d) show the results from the first 10 epochs using KL loss. Plots (b) and (e) depict performance after switching from KL loss to Cross Entropy loss, weighted by class distribution. Finally, plots (c) and (f) demonstrate performance when a regularization term (weight_decay) is applied with a value of 0.0001.
• Figure 4: Confusion matrices for vlassification model trained with lower or higher ResNet layers.
• Figure 5: Confusion matrices and precision-recall curves for the training and testing datasets of the emotion classifier trained with KL loss.