Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

Wei Zhang, Feng Qiu, Chen Liu, Lincheng Li, Heming Du, Tiancheng Guo, Xin Yu

TL;DR

This paper presents the method designs for a transformer-based feature fusion module to fully integrate emotional information provided by audio signals, visual images, and transcripts, offering high-quality expression features for the downstream tasks.

Abstract

Affective Behavior Analysis aims to facilitate technology emotionally smart, creating a world where devices can understand and react to our emotions as humans do. To comprehensively evaluate the authenticity and applicability of emotional behavior analysis techniques in natural environments, the 6th competition on Affective Behavior Analysis in-the-wild (ABAW) utilizes the Aff-Wild2, Hume-Vidmimic2, and C-EXPR-DB datasets to set up five competitive tracks, i.e., Valence-Arousal (VA) Estimation, Expression (EXPR) Recognition, Action Unit (AU) Detection, Compound Expression (CE) Recognition, and Emotional Mimicry Intensity (EMI) Estimation. In this paper, we present our method designs for the five tasks. Specifically, our design mainly includes three aspects: 1) Utilizing a transformer-based feature fusion module to fully integrate emotional information provided by audio signals, visual images, and transcripts, offering high-quality expression features for the downstream tasks. 2) To achieve high-quality facial feature representations, we employ Masked-Auto Encoder as the visual features extraction model and fine-tune it with our facial dataset. 3) Considering the complexity of the video collection scenes, we conduct a more detailed dataset division based on scene characteristics and train the classifier for each scene. Extensive experiments demonstrate the superiority of our designs.

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

TL;DR

This paper presents the method designs for a transformer-based feature fusion module to fully integrate emotional information provided by audio signals, visual images, and transcripts, offering high-quality expression features for the downstream tasks.

Abstract

Affective Behavior Analysis aims to facilitate technology emotionally smart, creating a world where devices can understand and react to our emotions as humans do. To comprehensively evaluate the authenticity and applicability of emotional behavior analysis techniques in natural environments, the 6th competition on Affective Behavior Analysis in-the-wild (ABAW) utilizes the Aff-Wild2, Hume-Vidmimic2, and C-EXPR-DB datasets to set up five competitive tracks, i.e., Valence-Arousal (VA) Estimation, Expression (EXPR) Recognition, Action Unit (AU) Detection, Compound Expression (CE) Recognition, and Emotional Mimicry Intensity (EMI) Estimation. In this paper, we present our method designs for the five tasks. Specifically, our design mainly includes three aspects: 1) Utilizing a transformer-based feature fusion module to fully integrate emotional information provided by audio signals, visual images, and transcripts, offering high-quality expression features for the downstream tasks. 2) To achieve high-quality facial feature representations, we employ Masked-Auto Encoder as the visual features extraction model and fine-tune it with our facial dataset. 3) Considering the complexity of the video collection scenes, we conduct a more detailed dataset division based on scene characteristics and train the classifier for each scene. Extensive experiments demonstrate the superiority of our designs.
Paper Structure (21 sections, 11 equations, 1 figure, 4 tables)

This paper contains 21 sections, 11 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Action Unit Detection
  4. Compound Expression Recognition
  5. Emotional Mimicry Intensity Estimation
  6. Expression Recognition
  7. Valence-arousal Estimation
  8. Method
  9. Feature Extraction Encoder
  10. Transformer-based Multi-modal Fusion
  11. Ensemble Learning
  12. Training Objectives
  13. Experiment
  14. Evaluation metrics
  15. Datasets
  16. ...and 6 more sections

Figures (1)

  • Figure 1: The overview of our proposed method. We first utilize the images in the facial image datasets to train the Image Encoder in a self-supervised manner, thus obtaining the visual feature $F_I$. Then we leverage the pre-trained audio encoder and text encoder to attain the audio feature $F_A$ and text feature $F_T$. Note that we only devise the text encoder for the EMI task. Subsequently, we concat these features and feed them into the Transformer Encoders. Here, we train these encoders on subsets divided based on background characteristics. Finally, we employ a voting strategy to attain the final results.