Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SUN Team's Contribution to ABAW 2024 Competition: Audio-visual Valence-Arousal Estimation and Expression Recognition

Denis Dresvyanskiy, Maxim Markitantov, Jiawei Yu, Peitong Li, Heysem Kaya, Alexey Karpov

TL;DR

This work investigates audiovisual deep learning approaches for emotion recognition in-the-wild problem and explores the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM), for video and audio modality, respectively.

Abstract

As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely 'in-the-wild' data. This work investigates audiovisual deep learning approaches for emotion recognition in-the-wild problem. We particularly explore the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM), for video and audio modality, respectively. We compare alternative temporal modeling and fusion strategies using the embeddings from these multi-stage trained modality-specific Deep Neural Networks (DNN). We report results on the AffWild2 dataset under Affective Behavior Analysis in-the-Wild 2024 (ABAW'24) challenge protocol.

SUN Team's Contribution to ABAW 2024 Competition: Audio-visual Valence-Arousal Estimation and Expression Recognition

TL;DR

This work investigates audiovisual deep learning approaches for emotion recognition in-the-wild problem and explores the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM), for video and audio modality, respectively.

Abstract

As emotions play a central role in human communication, automatic emotion recognition has attracted increasing attention in the last two decades. While multimodal systems enjoy high performances on lab-controlled data, they are still far from providing ecological validity on non-lab-controlled, namely 'in-the-wild' data. This work investigates audiovisual deep learning approaches for emotion recognition in-the-wild problem. We particularly explore the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM), for video and audio modality, respectively. We compare alternative temporal modeling and fusion strategies using the embeddings from these multi-stage trained modality-specific Deep Neural Networks (DNN). We report results on the AffWild2 dataset under Affective Behavior Analysis in-the-Wild 2024 (ABAW'24) challenge protocol.
Paper Structure (19 sections, 2 equations, 5 figures, 6 tables)

This paper contains 19 sections, 2 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Acoustic Emotion Recognition System
  4. Visual Emotion Recognition System
  5. Static Models
  6. Dynamic Models
  7. Fusion Schemes
  8. Experimental Setup
  9. Experimental Data
  10. Pre-training Data
  11. Data Preprocessing
  12. Audio
  13. Video
  14. Post-processing
  15. Experimental Results
  16. ...and 4 more sections

Figures (5)

  • Figure 1: The neural network architecture of modified frame-level FER models. n -- number of neurons, 8 for classification task and 2 for regression task. After generation of predictions, a softmax or tanh activation function is applied depending on the task type.
  • Figure 2: The architecture of Kernel ELM based dynamic emotion recognition model.
  • Figure 3: Pipeline and architecture of the transformer-based sequence to one video emotion recognition model. $W$ -- the temporal window size (in the number of frames), $N$ -- the number of neurons in the decision-making head (either 8 for classification or 2 for regression task).
  • Figure 4: Pipeline of the image preprocessing for the static models.
  • Figure 5: Pipeline of preprocessing of the video data for dynamic emotion recognition modeling.