Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations

Yu Liu, Hanlei Shi, Haoxun Li, Yuqing Sun, Yuxuan Ding, Linlin Gong, Leyuan Qu, Taihao Li

TL;DR

A unified model that detects per-utterance hotspots in text, audio, and video, fuses them with global features via Hotspot-Gated Fusion, and aligns modalities using a routed Mixture-of-Aligners; a cross-modal graph encodes conversational structure is presented.

Abstract

Emotion Recognition in Conversations (ERC) is hard because discriminative evidence is sparse, localized, and often asynchronous across modalities. We center ERC on emotion hotspots and present a unified model that detects per-utterance hotspots in text, audio, and video, fuses them with global features via Hotspot-Gated Fusion, and aligns modalities using a routed Mixture-of-Aligners; a cross-modal graph encodes conversational structure. This design focuses modeling on salient spans, mitigates misalignment, and preserves context. Experiments on standard ERC benchmarks show consistent gains over strong baselines, with ablations confirming the contributions of HGF and MoA. Our results point to a hotspot-centric view that can inform future multimodal learning, offering a new perspective on modality fusion in ERC.

Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations

TL;DR

A unified model that detects per-utterance hotspots in text, audio, and video, fuses them with global features via Hotspot-Gated Fusion, and aligns modalities using a routed Mixture-of-Aligners; a cross-modal graph encodes conversational structure is presented.

Abstract

Emotion Recognition in Conversations (ERC) is hard because discriminative evidence is sparse, localized, and often asynchronous across modalities. We center ERC on emotion hotspots and present a unified model that detects per-utterance hotspots in text, audio, and video, fuses them with global features via Hotspot-Gated Fusion, and aligns modalities using a routed Mixture-of-Aligners; a cross-modal graph encodes conversational structure. This design focuses modeling on salient spans, mitigates misalignment, and preserves context. Experiments on standard ERC benchmarks show consistent gains over strong baselines, with ablations confirming the contributions of HGF and MoA. Our results point to a hotspot-centric view that can inform future multimodal learning, offering a new perspective on modality fusion in ERC.

Paper Structure

This paper contains 12 sections, 9 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Overview
  4. Inputs and Hotspot-Gated Fusion
  5. MoA for Cross-modal Alignment
  6. Cross-Modal Graph Pathway
  7. Loss Function
  8. Experiments
  9. Experimental Setting
  10. Overall Results
  11. Ablation Study
  12. Conclusion

Figures (2)

  • Figure 1: The model locates multimodal emotion hotspots and employs gated fusion and MoA for cross-modal alignment, resolving temporal and semantic conflicts.
  • Figure 2: Per-utterance inputs from text (T), audio (A), and video (V) are first combined by HGF and then encoded into a shared representation. Two pathways operate in parallel: (i) MoA performs routed multi-expert cross-modal alignment and builds a modality memory; (ii) a graph models relational structure. The pathway outputs are concatenated and fed to a classifier. Implementation details are deferred to Section \ref{['sec:method']}.