Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Orthogonal Disentanglement with Projected Feature Alignment for Multimodal Emotion Recognition in Conversation

Xinyi Che, Wenbo Wang, Jian Guan, Qijun Zhao

TL;DR

The paper tackles MERC by addressing the gap in capturing modality-specific cues alongside shared semantics. It introduces OD-PFA, which disentangles unimodal features into shared and modality-specific components via an orthogonal constraint with reconstruction losses, and aligns shared features across modalities by projecting them into a common textual latent space with cross-modal consistency. Through extensive experiments on IEMOCAP and MELD, OD-PFA achieves state-of-the-art or competitive results across accuracy and weighted F1, illustrating the benefit of combining orthogonal disentanglement with projection-based alignment. This approach advances robust MERC by simultaneously preserving modality-specific information and enforcing cross-modal semantic coherence.

Abstract

Multimodal Emotion Recognition in Conversation (MERC) significantly enhances emotion recognition performance by integrating complementary emotional cues from text, audio, and visual modalities. While existing methods commonly utilize techniques such as contrastive learning and cross-attention mechanisms to align cross-modal emotional semantics, they typically overlook modality-specific emotional nuances like micro-expressions, tone variations, and sarcastic language. To overcome these limitations, we propose Orthogonal Disentanglement with Projected Feature Alignment (OD-PFA), a novel framework designed explicitly to capture both shared semantics and modality-specific emotional cues. Our approach first decouples unimodal features into shared and modality-specific components. An orthogonal disentanglement strategy (OD) enforces effective separation between these components, aided by a reconstruction loss to maintain critical emotional information from each modality. Additionally, a projected feature alignment strategy (PFA) maps shared features across modalities into a common latent space and applies a cross-modal consistency alignment loss to enhance semantic coherence. Extensive evaluations on widely-used benchmark datasets, IEMOCAP and MELD, demonstrate effectiveness of our proposed OD-PFA multimodal emotion recognition tasks, as compared with the state-of-the-art approaches.

Orthogonal Disentanglement with Projected Feature Alignment for Multimodal Emotion Recognition in Conversation

TL;DR

The paper tackles MERC by addressing the gap in capturing modality-specific cues alongside shared semantics. It introduces OD-PFA, which disentangles unimodal features into shared and modality-specific components via an orthogonal constraint with reconstruction losses, and aligns shared features across modalities by projecting them into a common textual latent space with cross-modal consistency. Through extensive experiments on IEMOCAP and MELD, OD-PFA achieves state-of-the-art or competitive results across accuracy and weighted F1, illustrating the benefit of combining orthogonal disentanglement with projection-based alignment. This approach advances robust MERC by simultaneously preserving modality-specific information and enforcing cross-modal semantic coherence.

Abstract

Multimodal Emotion Recognition in Conversation (MERC) significantly enhances emotion recognition performance by integrating complementary emotional cues from text, audio, and visual modalities. While existing methods commonly utilize techniques such as contrastive learning and cross-attention mechanisms to align cross-modal emotional semantics, they typically overlook modality-specific emotional nuances like micro-expressions, tone variations, and sarcastic language. To overcome these limitations, we propose Orthogonal Disentanglement with Projected Feature Alignment (OD-PFA), a novel framework designed explicitly to capture both shared semantics and modality-specific emotional cues. Our approach first decouples unimodal features into shared and modality-specific components. An orthogonal disentanglement strategy (OD) enforces effective separation between these components, aided by a reconstruction loss to maintain critical emotional information from each modality. Additionally, a projected feature alignment strategy (PFA) maps shared features across modalities into a common latent space and applies a cross-modal consistency alignment loss to enhance semantic coherence. Extensive evaluations on widely-used benchmark datasets, IEMOCAP and MELD, demonstrate effectiveness of our proposed OD-PFA multimodal emotion recognition tasks, as compared with the state-of-the-art approaches.

Paper Structure

This paper contains 17 sections, 16 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Multi-Modality Feature Extraction
  4. Unimodal Feature Extraction
  5. Modality-Shared and Specific Feature Extraction
  6. Modality-Shared and Specific Feature Extraction
  7. Orthogonal Feature Disentanglement
  8. Cross-Modal Feature Projection and Alignment
  9. Shared Space Feature Projection
  10. Cross-Modal Consistency Alignment
  11. Emotion Recognition
  12. Experiments
  13. Experimental Setup
  14. Performance Comparison
  15. Ablation Study
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Overall framework of our proposed OD-PFA for multi-modal emotion recognition in conversation. It consists of four components: Multi-Modality Feature Extraction, Orthogonal Feature Disentanglement, Cross-Modal Feature Projection and Alignment, and the Emotion Recognition.