Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

Haoxiang Shi, Xulong Zhang, Ning Cheng, Yong Zhang, Jun Yu, Jing Xiao, Jianzong Wang

TL;DR

This work tackles emotion recognition in conversations by exploiting cross-modal information with a vector-based joint fusion mechanism and supervised inter-class contrastive learning. Text and audio are modeled by pre-trained transformers (RoBERTa, ViT/LTrans) and fused through trainable joint vectors, producing joint fusion features that feed dual emotion classifiers. An inter-class contrastive loss further enforces separation between different emotion categories while tightening intra-class variability, addressing data imbalance. Experiments on IEMOCAP and MELD show competitive or superior performance across single-modal, text-audio, and multi-modal settings, highlighting the approach's effectiveness and practical potential for empathetic dialogue systems.

Abstract

The purpose of emotion recognition in conversation (ERC) is to identify the emotion category of an utterance based on contextual information. Previous ERC methods relied on simple connections for cross-modal fusion and ignored the information differences between modalities, resulting in the model being unable to focus on modality-specific emotional information. At the same time, the shared information between modalities was not processed to generate emotions. Information redundancy problem. To overcome these limitations, we propose a cross-modal fusion emotion prediction network based on vector connections. The network mainly includes two stages: the multi-modal feature fusion stage based on connection vectors and the emotion classification stage based on fused features. Furthermore, we design a supervised inter-class contrastive learning module based on emotion labels. Experimental results confirm the effectiveness of the proposed method, demonstrating excellent performance on the IEMOCAP and MELD datasets.

Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

TL;DR

This work tackles emotion recognition in conversations by exploiting cross-modal information with a vector-based joint fusion mechanism and supervised inter-class contrastive learning. Text and audio are modeled by pre-trained transformers (RoBERTa, ViT/LTrans) and fused through trainable joint vectors, producing joint fusion features that feed dual emotion classifiers. An inter-class contrastive loss further enforces separation between different emotion categories while tightening intra-class variability, addressing data imbalance. Experiments on IEMOCAP and MELD show competitive or superior performance across single-modal, text-audio, and multi-modal settings, highlighting the approach's effectiveness and practical potential for empathetic dialogue systems.

Abstract

The purpose of emotion recognition in conversation (ERC) is to identify the emotion category of an utterance based on contextual information. Previous ERC methods relied on simple connections for cross-modal fusion and ignored the information differences between modalities, resulting in the model being unable to focus on modality-specific emotional information. At the same time, the shared information between modalities was not processed to generate emotions. Information redundancy problem. To overcome these limitations, we propose a cross-modal fusion emotion prediction network based on vector connections. The network mainly includes two stages: the multi-modal feature fusion stage based on connection vectors and the emotion classification stage based on fused features. Furthermore, we design a supervised inter-class contrastive learning module based on emotion labels. Experimental results confirm the effectiveness of the proposed method, demonstrating excellent performance on the IEMOCAP and MELD datasets.
Paper Structure (16 sections, 7 equations, 3 figures, 3 tables)

This paper contains 16 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Task Definition
  4. Overview
  5. Joint-based Multi-modal Fusion
  6. Emotion Recognition
  7. Inter-class Contrastive Learning
  8. Experiments
  9. Datasets and Baselines
  10. Implementation Details
  11. Results Analysis
  12. Ablation Study
  13. Fusion Method Comparison
  14. Parameter Analysis
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: The architecture of our model. The left and right sides represent text and audio inputs, respectively. By combining text and audio features, the cross-modal joint vector $v_j$ is used for multi-modal information fusion to extract the fused feature representation, and finally, perform emotion classification. Simultaneously, different colors in the contrast learning module represent different emotional category samples.
  • Figure 2: Accuracy confusion matrix on two datasets.
  • Figure 3: Model performance under different numbers of JF blocks and different lengths of joint vector.