Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Space Synergy: A Unified Framework for Multimodal Emotion Recognition in Conversation

Xiaosen Lyu, Jiayu Xiong, Yuren Chen, Wanlong Wang, Xiaoqing Dai, Jing Wang

TL;DR

This work introduces Cross-Space Synergy (CSS) for Multimodal Emotion Recognition in Conversation, coupling a high-order, low-rank SPF representation fusion with Pareto-gradient-based optimization (PGM) to address gradient conflicts and unstable training. The framework combines a two-stage representation encoder with modality-specific projections and gated cross-modal interactions, enabling expressive yet stable cross-modal fusion. Through experiments on IEMOCAP and MELD, CSS achieves state-of-the-art accuracy and training stability while maintaining efficiency, demonstrated by ablation analyses on SPF, PGM, and modality contributions. The results suggest that coordinating representation-space fusion with gradient-space optimization is crucial for robust, scalable MERC systems.

Abstract

Multimodal Emotion Recognition in Conversation (MERC) aims to predict speakers' emotions by integrating textual, acoustic, and visual cues. Existing approaches either struggle to capture complex cross-modal interactions or experience gradient conflicts and unstable training when using deeper architectures. To address these issues, we propose Cross-Space Synergy (CSS), which couples a representation component with an optimization component. Synergistic Polynomial Fusion (SPF) serves the representation role, leveraging low-rank tensor factorization to efficiently capture high-order cross-modal interactions. Pareto Gradient Modulator (PGM) serves the optimization role, steering updates along Pareto-optimal directions across competing objectives to alleviate gradient conflicts and improve stability. Experiments show that CSS outperforms existing representative methods on IEMOCAP and MELD in both accuracy and training stability, demonstrating its effectiveness in complex multimodal scenarios.

Cross-Space Synergy: A Unified Framework for Multimodal Emotion Recognition in Conversation

TL;DR

This work introduces Cross-Space Synergy (CSS) for Multimodal Emotion Recognition in Conversation, coupling a high-order, low-rank SPF representation fusion with Pareto-gradient-based optimization (PGM) to address gradient conflicts and unstable training. The framework combines a two-stage representation encoder with modality-specific projections and gated cross-modal interactions, enabling expressive yet stable cross-modal fusion. Through experiments on IEMOCAP and MELD, CSS achieves state-of-the-art accuracy and training stability while maintaining efficiency, demonstrated by ablation analyses on SPF, PGM, and modality contributions. The results suggest that coordinating representation-space fusion with gradient-space optimization is crucial for robust, scalable MERC systems.

Abstract

Multimodal Emotion Recognition in Conversation (MERC) aims to predict speakers' emotions by integrating textual, acoustic, and visual cues. Existing approaches either struggle to capture complex cross-modal interactions or experience gradient conflicts and unstable training when using deeper architectures. To address these issues, we propose Cross-Space Synergy (CSS), which couples a representation component with an optimization component. Synergistic Polynomial Fusion (SPF) serves the representation role, leveraging low-rank tensor factorization to efficiently capture high-order cross-modal interactions. Pareto Gradient Modulator (PGM) serves the optimization role, steering updates along Pareto-optimal directions across competing objectives to alleviate gradient conflicts and improve stability. Experiments show that CSS outperforms existing representative methods on IEMOCAP and MELD in both accuracy and training stability, demonstrating its effectiveness in complex multimodal scenarios.

Paper Structure

This paper contains 16 sections, 16 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Emotion Recognition in Conversation
  4. Multimodal Fusion Methods
  5. Multi-Objective Optimization Strategies
  6. Methodology
  7. Overview
  8. Representation Encoding
  9. Representation-Space Fusion
  10. Gradient-Space Optimization
  11. Experiments
  12. Experimental Setup
  13. Baseline Details
  14. Interpretation and Discussion
  15. Ablation Studies
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of the proposed CSS framework, which integrates two-stage representation encoding, high-order multimodal fusion (SPF), and multi-objective optimization (PGM).
  • Figure 2: Training loss curves on IEMOCAP with and without PGM. The left two plots show multimodal loss $\mathcal{L}_1$ and unimodal losses $\mathcal{L}_2^{(m)}$; the right show KL losses $\mathcal{L}_3^{(m)}$. PGM improves convergence and reduces variance, especially for KL losses.