Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning

TL;DR

This work tackles emotion conflicts across modalities in multimodal emotion reasoning by introducing CA-MER, a benchmark with video-aligned, audio-aligned, and consistent subsets to reveal modality biases in current MLLMs. It analyzes why audio signals dominate during conflicts and introduces MoSEAR, a two-part framework combining Modality-Specific Experts (MoSE) and Attention Reallocation (AR) to balance multimodal contributions without sacrificing cross-modality performance. MoSEAR demonstrates state-of-the-art results on CA-MER and related datasets (MER2023, EMER, DFEW), reduces the performance gap between modalities, and generalizes to open-ended reasoning with human-aligned improvements. The work also provides practical insights into token imbalance as a root cause and offers a low-overhead, training-efficient solution suitable for real-world deployment in emotion-aware AI systems.

Abstract

Despite their strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook the scenarios involving emotion conflicts, where emotional cues from different modalities are inconsistent. To fill this gap, we first introduce CA-MER, a new benchmark designed to examine MLLMs under realistic emotion conflicts. It consists of three subsets: video-aligned, audio-aligned, and consistent, where only one or all modalities reflect the true emotion. However, evaluations on our CA-MER reveal that current state-of-the-art emotion MLLMs systematically over-rely on audio signal during emotion conflicts, neglecting critical cues from visual modality. To mitigate this bias, we propose MoSEAR, a parameter-efficient framework that promotes balanced modality integration. MoSEAR consists of two modules: (1)MoSE, modality-specific experts with a regularized gating mechanism that reduces modality bias in the fine-tuning heads; and (2)AR, an attention reallocation mechanism that rebalances modality contributions in frozen backbones during inference. Our framework offers two key advantages: it mitigates emotion conflicts and improves performance on consistent samples-without incurring a trade-off between audio and visual modalities. Experiments on multiple benchmarks-including MER2023, EMER, DFEW, and our CA-MER-demonstrate that MoSEAR achieves state-of-the-art performance, particularly under modality conflict conditions.

Figures (5)

• Figure 1: Example of an emotion conflict case with reasoning outputs from Emotion-LLaMA and our MoSEAR. (a) A visual-aligned sample in which the character’s facial expression conveys a clear sense of disappointment. (b) Our MoSEAR provides a correct emotion reasoning, while Emotion-LLaMA produces an incorrect one under emotion conflict.
• Figure 2: of CA-MER construction. (a) The three-stage construction process of our CA-MER dataset. (b) Example samples from the three subsets: video-aligned, consistent, and audio-aligned. Video cues are in red, and audio cues are in blue.
• Figure 3: Analyses of modality bias in emotion conflicts
• Figure 4: Illustration of modality-specific experts.
• Figure 5: Impact of the hyper-parameter $\epsilon$.