Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation

Taeyang Yun, Hyunkuk Lim, Jeonghwan Lee, Min Song

TL;DR

The paper addresses Emotion Recognition in Conversation (ERC) by leveraging multimodal cues while acknowledging the varying contributions of each modality. It introduces TelME, a framework that distills knowledge from a strong text teacher into weaker audio and visual student encoders and fuses modalities via an attention-based shifting mechanism. Empirical results on MELD and IEMOCAP demonstrate state-of-the-art performance on MELD, with ablations confirming the efficacy of cross-modal knowledge distillation and the shifting fusion strategy. The work highlights a practical path for improving ERC systems through cross-modal transfer, especially when textual signals are dominant, while also identifying limitations related to visual cues and dataset imbalance.

Abstract

Emotion Recognition in Conversation (ERC) plays a crucial role in enabling dialogue systems to effectively respond to user requests. The emotions in a conversation can be identified by the representations from various modalities, such as audio, visual, and text. However, due to the weak contribution of non-verbal modalities to recognize emotions, multimodal ERC has always been considered a challenging task. In this paper, we propose Teacher-leading Multimodal fusion network for ERC (TelME). TelME incorporates cross-modal knowledge distillation to transfer information from a language model acting as the teacher to the non-verbal students, thereby optimizing the efficacy of the weak modalities. We then combine multimodal features using a shifting fusion approach in which student networks support the teacher. TelME achieves state-of-the-art performance in MELD, a multi-speaker conversation dataset for ERC. Finally, we demonstrate the effectiveness of our components through additional experiments.

TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation

TL;DR

The paper addresses Emotion Recognition in Conversation (ERC) by leveraging multimodal cues while acknowledging the varying contributions of each modality. It introduces TelME, a framework that distills knowledge from a strong text teacher into weaker audio and visual student encoders and fuses modalities via an attention-based shifting mechanism. Empirical results on MELD and IEMOCAP demonstrate state-of-the-art performance on MELD, with ablations confirming the efficacy of cross-modal knowledge distillation and the shifting fusion strategy. The work highlights a practical path for improving ERC systems through cross-modal transfer, especially when textual signals are dominant, while also identifying limitations related to visual cues and dataset imbalance.

Abstract

Emotion Recognition in Conversation (ERC) plays a crucial role in enabling dialogue systems to effectively respond to user requests. The emotions in a conversation can be identified by the representations from various modalities, such as audio, visual, and text. However, due to the weak contribution of non-verbal modalities to recognize emotions, multimodal ERC has always been considered a challenging task. In this paper, we propose Teacher-leading Multimodal fusion network for ERC (TelME). TelME incorporates cross-modal knowledge distillation to transfer information from a language model acting as the teacher to the non-verbal students, thereby optimizing the efficacy of the weak modalities. We then combine multimodal features using a shifting fusion approach in which student networks support the teacher. TelME achieves state-of-the-art performance in MELD, a multi-speaker conversation dataset for ERC. Finally, we demonstrate the effectiveness of our components through additional experiments.
Paper Structure (28 sections, 19 equations, 7 figures, 11 tables)

This paper contains 28 sections, 19 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Emotion Recognition in Conversation
  4. Knowledge Distillation
  5. Method
  6. Problem Statement
  7. TelME
  8. Model Overview
  9. Feature Extraction
  10. Knowledge Distillation
  11. Attention-based modality Shifting Fusion
  12. Experiments
  13. Datasets
  14. Experiment Settings
  15. Main Results
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Examples of multimodal ERC. Even the same "Okay" answer varies depending on the conversation situation and captures emotions in various modalities.
  • Figure 2: Unimodal Performance on MELD dataset
  • Figure 3: The overview of TelME
  • Figure 4: Attention-based modality Shifting Fusion
  • Figure 5: Individual performance of audio and visual modalities according to knowledge distillation type.
  • ...and 2 more figures