Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emotional Cues Extraction and Fusion for Multi-modal Emotion Prediction and Recognition in Conversation

Haoxiang Shi, Ziqi Liang, Jun Yu

TL;DR

The paper addresses Emotion Prediction in Conversation (EPC) by incorporating fine-grained word-level emotional cues and modality-aware fusion. It introduces KWRT to capture word-level relationships in text and a Prosody Enhancement (PE) module for audio, followed by a two-step fusion (TMF) that leverages mel-spectrogram spectral-domain features. Empirical results on IEMOCAP and MELD show state-of-the-art EPC performance and competitive ERC performance, with ablations confirming the value of each component. The approach offers robust, multi-modal emotion perception and improved integration for conversational AI, with clear multi-task applicability to ERC.

Abstract

Emotion Prediction in Conversation (EPC) aims to forecast the emotions of forthcoming utterances by utilizing preceding dialogues. Previous EPC approaches relied on simple context modeling for emotion extraction, overlooking fine-grained emotion cues at the word level. Additionally, prior works failed to account for the intrinsic differences between modalities, resulting in redundant information. To overcome these limitations, we propose an emotional cues extraction and fusion network, which consists of two stages: a modality-specific learning stage that utilizes word-level labels and prosody learning to construct emotion embedding spaces for each modality, and a two-step fusion stage for integrating multi-modal features. Moreover, the emotion features extracted by our model are also applicable to the Emotion Recognition in Conversation (ERC) task. Experimental results validate the efficacy of the proposed method, demonstrating superior performance on both IEMOCAP and MELD datasets.

Emotional Cues Extraction and Fusion for Multi-modal Emotion Prediction and Recognition in Conversation

TL;DR

The paper addresses Emotion Prediction in Conversation (EPC) by incorporating fine-grained word-level emotional cues and modality-aware fusion. It introduces KWRT to capture word-level relationships in text and a Prosody Enhancement (PE) module for audio, followed by a two-step fusion (TMF) that leverages mel-spectrogram spectral-domain features. Empirical results on IEMOCAP and MELD show state-of-the-art EPC performance and competitive ERC performance, with ablations confirming the value of each component. The approach offers robust, multi-modal emotion perception and improved integration for conversational AI, with clear multi-task applicability to ERC.

Abstract

Emotion Prediction in Conversation (EPC) aims to forecast the emotions of forthcoming utterances by utilizing preceding dialogues. Previous EPC approaches relied on simple context modeling for emotion extraction, overlooking fine-grained emotion cues at the word level. Additionally, prior works failed to account for the intrinsic differences between modalities, resulting in redundant information. To overcome these limitations, we propose an emotional cues extraction and fusion network, which consists of two stages: a modality-specific learning stage that utilizes word-level labels and prosody learning to construct emotion embedding spaces for each modality, and a two-step fusion stage for integrating multi-modal features. Moreover, the emotion features extracted by our model are also applicable to the Emotion Recognition in Conversation (ERC) task. Experimental results validate the efficacy of the proposed method, demonstrating superior performance on both IEMOCAP and MELD datasets.
Paper Structure (14 sections, 6 equations, 3 figures, 3 tables)

This paper contains 14 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Task Definition
  4. Overview
  5. Knowledge-based Word Relation Tagging (KWRT)
  6. Prosody Enhancement (PE)
  7. Two-step Multi-modal Fusion Layer (TMF)
  8. Experiments
  9. Datasets and Baselines
  10. Implementation
  11. Results analysis
  12. Ablation Study
  13. Conclusion
  14. Acknowledgement

Figures (3)

  • Figure 1: The left side is the main framework of our model. (a) multi-modal fusion module, the green part represents the pre-trained language transformer layer, the yellow part represents the pre-trained vision transformer layer, the pink part is trainable, and other parts are frozen. It is worth noting that, in (b), we first initialize the the word-level relation matrix $M_{rel}$ with 0. For each word pair, if a relation exists, the importance is increased by 1, with a maximum upper limit of 3.
  • Figure 2: A tagging example with KWRT module.
  • Figure 3: Accuracy confusion matrix on different tasks.