Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment

Xuechen Wang, Shiwan Zhao, Haoqin Sun, Hui Wang, Jiaming Zhou, Yong Qin

TL;DR

The paper tackles multimodal emotion recognition (MER) from speech and text, focusing on robust cross-modal alignment amid heterogeneity and emotional ambiguity. It introduces the Multi-Granularity Cross-Modal Alignment (MGCMA) framework, which combines distribution-based, token-based, and instance-based alignment alongside a Wav2vec2.0-BERT backbone. Distribution-based alignment constructs Gaussian representations and uses the distance $W_2$ with a similarity $Sim$ to guide cross-modal alignment; token-based alignment applies self- and cross-attention for fine-grained matching; instance-based alignment employs contrastive learning to reinforce correct speech-text mappings, all optimized with a final cross-entropy loss. On IEMOCAP, MGCMA achieves WA $78.87\%$ and UA $80.24\%$, outperforming prior SOTA methods and demonstrating the value of multi-granularity cross-modal alignment for MER. This approach advances practical MER by leveraging multi-level information to better handle emotional complexity and cross-modal dependencies.

Abstract

Multimodal emotion recognition (MER), leveraging speech and text, has emerged as a pivotal domain within human-computer interaction, demanding sophisticated methods for effective multimodal integration. The challenge of aligning features across these modalities is significant, with most existing approaches adopting a singular alignment strategy. Such a narrow focus not only limits model performance but also fails to address the complexity and ambiguity inherent in emotional expressions. In response, this paper introduces a Multi-Granularity Cross-Modal Alignment (MGCMA) framework, distinguished by its comprehensive approach encompassing distribution-based, instance-based, and token-based alignment modules. This framework enables a multi-level perception of emotional information across modalities. Our experiments on IEMOCAP demonstrate that our proposed method outperforms current state-of-the-art techniques.

Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment

TL;DR

The paper tackles multimodal emotion recognition (MER) from speech and text, focusing on robust cross-modal alignment amid heterogeneity and emotional ambiguity. It introduces the Multi-Granularity Cross-Modal Alignment (MGCMA) framework, which combines distribution-based, token-based, and instance-based alignment alongside a Wav2vec2.0-BERT backbone. Distribution-based alignment constructs Gaussian representations and uses the distance with a similarity to guide cross-modal alignment; token-based alignment applies self- and cross-attention for fine-grained matching; instance-based alignment employs contrastive learning to reinforce correct speech-text mappings, all optimized with a final cross-entropy loss. On IEMOCAP, MGCMA achieves WA and UA , outperforming prior SOTA methods and demonstrating the value of multi-granularity cross-modal alignment for MER. This approach advances practical MER by leveraging multi-level information to better handle emotional complexity and cross-modal dependencies.

Abstract

Multimodal emotion recognition (MER), leveraging speech and text, has emerged as a pivotal domain within human-computer interaction, demanding sophisticated methods for effective multimodal integration. The challenge of aligning features across these modalities is significant, with most existing approaches adopting a singular alignment strategy. Such a narrow focus not only limits model performance but also fails to address the complexity and ambiguity inherent in emotional expressions. In response, this paper introduces a Multi-Granularity Cross-Modal Alignment (MGCMA) framework, distinguished by its comprehensive approach encompassing distribution-based, instance-based, and token-based alignment modules. This framework enables a multi-level perception of emotional information across modalities. Our experiments on IEMOCAP demonstrate that our proposed method outperforms current state-of-the-art techniques.
Paper Structure (14 sections, 6 equations, 3 figures, 3 tables)

This paper contains 14 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Feature Extractor
  4. Distribution-based Alignment Module
  5. Distribution Constructor
  6. Distribution-level Contrastive Learning
  7. Token-based Alignment Module
  8. Instance-based Alignment Module
  9. Experiments
  10. Dataset
  11. Experimental Settings
  12. Comparison with SOTA Approaches
  13. Ablation Studies and Analysis
  14. Conclusions

Figures (3)

  • Figure 1: Overview of our proposed Multi-Granularity Cross-Modal Alignment (MGCMA) framework which comprises distribution-based, token-based, instance-based alignment modules, and a feature extractor.
  • Figure 2: The structure of Distribution Constructor. Activation layers and normalization layers are omitted in the diagram.
  • Figure 3: Visualization of the representations with different alignment strategies, where darker colors indicate speech features and lighter colors indicate text features.