Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild

Kateryna Chumachenko, Alexandros Iosifidis, Moncef Gabbouj

TL;DR

This work tackles robust multimodal dynamic facial expression recognition in-the-wild by reusing publicly available unimodal SSL-pretrained encoders instead of relying on large-scale multimodal pre-training. It introduces progressive prompt learning to bridge intra-modality gaps, Fusion Bottleneck blocks for cross-modal alignment, and a Multimodal Temporal Transformer to capture temporal dynamics, while keeping the unimodal backbones frozen. The approach achieves state-of-the-art performance on two benchmarks, DFEW and MAFW, outperforming recent multimodal and temporal adaptation methods with fewer trainable parameters. The findings demonstrate that careful, lightweight adaptation of unimodal models can effectively leverage multimodal cues for DFER without extensive multimodal pre-training, enabling practical deployment and potential extension to additional modalities.

Abstract

Dynamic Facial Expression Recognition (DFER) has received significant interest in the recent years dictated by its pivotal role in enabling empathic and human-compatible technologies. Achieving robustness towards in-the-wild data in DFER is particularly important for real-world applications. One of the directions aimed at improving such models is multimodal emotion recognition based on audio and video data. Multimodal learning in DFER increases the model capabilities by leveraging richer, complementary data representations. Within the field of multimodal DFER, recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. Another line of research has focused on adapting pre-trained static models for DFER. In this work, we propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders. We identify main challenges associated with this task, namely, intra-modality adaptation, cross-modal alignment, and temporal adaptation, and propose solutions to each of them. As a result, we demonstrate improvement over current state-of-the-art on two popular DFER benchmarks, namely DFEW and MFAW.

MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild

TL;DR

This work tackles robust multimodal dynamic facial expression recognition in-the-wild by reusing publicly available unimodal SSL-pretrained encoders instead of relying on large-scale multimodal pre-training. It introduces progressive prompt learning to bridge intra-modality gaps, Fusion Bottleneck blocks for cross-modal alignment, and a Multimodal Temporal Transformer to capture temporal dynamics, while keeping the unimodal backbones frozen. The approach achieves state-of-the-art performance on two benchmarks, DFEW and MAFW, outperforming recent multimodal and temporal adaptation methods with fewer trainable parameters. The findings demonstrate that careful, lightweight adaptation of unimodal models can effectively leverage multimodal cues for DFER without extensive multimodal pre-training, enabling practical deployment and potential extension to additional modalities.

Abstract

Dynamic Facial Expression Recognition (DFER) has received significant interest in the recent years dictated by its pivotal role in enabling empathic and human-compatible technologies. Achieving robustness towards in-the-wild data in DFER is particularly important for real-world applications. One of the directions aimed at improving such models is multimodal emotion recognition based on audio and video data. Multimodal learning in DFER increases the model capabilities by leveraging richer, complementary data representations. Within the field of multimodal DFER, recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. Another line of research has focused on adapting pre-trained static models for DFER. In this work, we propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders. We identify main challenges associated with this task, namely, intra-modality adaptation, cross-modal alignment, and temporal adaptation, and propose solutions to each of them. As a result, we demonstrate improvement over current state-of-the-art on two popular DFER benchmarks, namely DFEW and MFAW.
Paper Structure (18 sections, 5 equations, 2 figures, 7 tables)

This paper contains 18 sections, 5 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dynamic Facial Expression Recognition in-the-wild
  4. Adaptation of pre-trained models
  5. Methodology
  6. Formulation
  7. Progressive prompt learning
  8. Modality fusion bottlenecks
  9. Temporal fusion
  10. Experiments
  11. Datasets
  12. Experimental setup
  13. Results
  14. Comparison of temporal adaptation approaches
  15. Comparison of fusion approaches
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Schematic description of MMA-DFER: Two pre-trained frozen MAE encoders are joined by Fusion Bottleneck for modality alignment, followed by joint adaptation module and Multimodal Temporal Transformer. Learnable prompts in each modality independently handle intra-modality gaps between pre-training and downstream data.
  • Figure 2: Fusion bottleneck. The features of each modality are first compressed to a low-dimensional representation, aggregated to obtain global representation per sequence, then fused with the complementary modality, and the joint representation is expanded back to the original space, and added to the initial features via a gating mechanism.