Ada2I: Enhancing Modality Balance for Multimodal Conversational Emotion Recognition

TL;DR

Ada2I targets modality imbalance in multimodal ERC by introducing two inseparable modules: AFW for feature-level balancing and AMW for modality-level balancing, guided by a tensor-ring interaction framework. It refines the disparity ratio to supervise training across all three modalities (text, audio, visual) and optimizes a joint objective $\,\mathcal{L}_{main} = \mathcal{L}_{modal} + \mathcal{L}_{feature} + \mathcal{L}_{cls}$ with additional gradient modulation and noise to stabilize learning. Empirically, Ada2I achieves state-of-the-art performance on IEMOCAP, MELD, and CMU-MOSEI, with notable gains on weaker modality pairs and reduced gaps between full-triple-modality and pairwise fusion, demonstrating improved balance and robustness. The approach advances practical ERC by ensuring weaker modalities contribute meaningfully during training, enabling more reliable emotion recognition in conversations.

Abstract

Multimodal Emotion Recognition in Conversations (ERC) is a typical multimodal learning task in exploiting various data modalities concurrently. Prior studies on effective multimodal ERC encounter challenges in addressing modality imbalances and optimizing learning across modalities. Dealing with these problems, we present a novel framework named Ada2I, which consists of two inseparable modules namely Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) for feature-level and modality-level balancing respectively via leveraging both Inter- and Intra-modal interactions. Additionally, we introduce a refined disparity ratio as part of our training optimization strategy, a simple yet effective measure to assess the overall discrepancy of the model's learning process when handling multiple modalities simultaneously. Experimental results validate the effectiveness of Ada2I with state-of-the-art performance compared to baselines on three benchmark datasets, particularly in addressing modality imbalances.

Figures (6)

• Figure 1: (a) Weighted F1 scores for the multimodal setting (T+A+V) compared with each unimodal encoder, and (b) batch-average unimodal-logit scores.
• Figure 2: Illustration of Ada2I framework
• Figure 3: Linear Transform block to compute core tensor.
• Figure 4: Performance gap visualizations between the multimodal setting (T+A+V) and pair-wise modality combinations are evaluated using the W-F1 metric across the IEMOCAP and MELD datasets.
• Figure 5: The change of the discrepancy ratio $\rho^t, \rho^a, \rho^v$ on the IEMOCAP and MELD datasets during training, along with various ablation tests including without AMW and without AFW, are compared to the Ada2I model.