Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Qifei Li, Yingming Gao, Yuhua Wen, Cong Wang, Ya Li

TL;DR

Foal-Net tackles multimodal emotion recognition by introducing two auxiliary tasks: AVEL, a contrastive alignment between audio and video embeddings, and MEM, a cross-modal emotion-label matching task with hard negatives. The fusion backbone relies on a two-layer multi-head cross-attention module to integrate aligned modalities, supervised by the emotion classification loss $L_{ce}$ along with $L_a$ and $L_m$. On the IEMOCAP corpus with five-fold leave-one-session-out evaluation, Foal-Net achieves state-of-the-art UA of $80.10\%$ and WA of $79.45\%$, outperforming prior methods and illustrating the necessity of alignment before fusion. The results also show CLIP-based video embeddings offer advantages over facial-only features, and the auxiliary tasks enhance robustness of multimodal fusion, implying broader applicability including potential extensions to text modality.

Abstract

To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after alignment, called Foal-Net. The framework is designed to enhance the effectiveness of modality fusion and includes two auxiliary tasks: audio-video emotion alignment (AVEL) and cross-modal emotion label matching (MEM). First, AVEL achieves alignment of emotional information in audio-video representations through contrastive learning. Then, a modal fusion network integrates the aligned features. Meanwhile, MEM assesses whether the emotions of the current sample pair are the same, providing assistance for modal information fusion and guiding the model to focus more on emotional information. The experimental results conducted on IEMOCAP corpus show that Foal-Net outperforms the state-of-the-art methods and emotion alignment is necessary before modal fusion.

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

TL;DR

Foal-Net tackles multimodal emotion recognition by introducing two auxiliary tasks: AVEL, a contrastive alignment between audio and video embeddings, and MEM, a cross-modal emotion-label matching task with hard negatives. The fusion backbone relies on a two-layer multi-head cross-attention module to integrate aligned modalities, supervised by the emotion classification loss along with and . On the IEMOCAP corpus with five-fold leave-one-session-out evaluation, Foal-Net achieves state-of-the-art UA of and WA of , outperforming prior methods and illustrating the necessity of alignment before fusion. The results also show CLIP-based video embeddings offer advantages over facial-only features, and the auxiliary tasks enhance robustness of multimodal fusion, implying broader applicability including potential extensions to text modality.

Abstract

To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after alignment, called Foal-Net. The framework is designed to enhance the effectiveness of modality fusion and includes two auxiliary tasks: audio-video emotion alignment (AVEL) and cross-modal emotion label matching (MEM). First, AVEL achieves alignment of emotional information in audio-video representations through contrastive learning. Then, a modal fusion network integrates the aligned features. Meanwhile, MEM assesses whether the emotions of the current sample pair are the same, providing assistance for modal information fusion and guiding the model to focus more on emotional information. The experimental results conducted on IEMOCAP corpus show that Foal-Net outperforms the state-of-the-art methods and emotion alignment is necessary before modal fusion.
Paper Structure (11 sections, 16 equations, 2 figures, 2 tables)

This paper contains 11 sections, 16 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Audio-Video Emotion Aligning
  4. Audio-Video Emotion Matching
  5. Modal Fusion Module
  6. Experiments and Results
  7. Experimental Setup
  8. Performance Comparison with Previous Methods
  9. Experiments and Analysis
  10. Conclusions
  11. Acknowledgements

Figures (2)

  • Figure 1: An overview of Foal-Net architecture, which mainly consists of two auxiliary tasks (AVEL, MEM), and modal fusion module.
  • Figure 2: The example for AVEL. The horizontal and vertical axis represents the video and audio modality respectively. As indicated in the subplot on the left, the value is 1 when their emotional labels are same; otherwise, it is 0. $CL$ denotes contrastive loss.