Calibrating Multimodal Consensus for Emotion Recognition

TL;DR

This work tackles semantic conflicts and text-dominated bias in multimodal emotion recognition by introducing Calibrated Multimodal Consensus (CMC), a two-stage framework comprised of a Pseudo Label Generation Module (PLGM), a Parameter-free Fusion Module (PFM), and a Multimodal Consensus Router (MCR). Unimodal pretraining with pseudo labels and supervised contrastive alignment establish robust modality representations, after which PFM preserves modality semantics during fusion and MCR dynamically calibrates modality weights to form a reliable consensus. Empirical results on CH-SIMS, CH-SIMS v2, CMU-MOSI, and CMU-MOSEI show that CMC achieves competitive or superior performance, with clear advantages on datasets exhibiting semantic inconsistencies; ablations confirm the necessity of all three modules. The approach is lightweight relative to large language-model–based baselines and demonstrates strong robustness to modality quality, suggesting practical impact for real-world MER systems and guiding future work toward integrating Multimodal Large Language Models and gradient-guided pseudo-labeling.

Abstract

In recent years, Multimodal Emotion Recognition (MER) has made substantial progress. Nevertheless, most existing approaches neglect the semantic inconsistencies that may arise across modalities, such as conflicting emotional cues between text and visual inputs. Besides, current methods are often dominated by the text modality due to its strong representational capacity, which can compromise recognition accuracy. To address these challenges, we propose a model termed Calibrated Multimodal Consensus (CMC). CMC introduces a Pseudo Label Generation Module (PLGM) to produce pseudo unimodal labels, enabling unimodal pretraining in a self-supervised fashion. It then employs a Parameter-free Fusion Module (PFM) and a Multimodal Consensus Router (MCR) for multimodal finetuning, thereby mitigating text dominance and guiding the fusion process toward a more reliable consensus. Experimental results demonstrate that CMC achieves performance on par with or superior to state-of-the-art methods across four datasets, CH-SIMS, CH-SIMS v2, CMU-MOSI, and CMU-MOSEI, and exhibits notable advantages in scenarios with semantic inconsistencies on CH-SIMS and CH-SIMS v2. The implementation of this work is publicly accessible at https://github.com/gw-zhong/CMC.

Figures (7)

• Figure 1: Two examples of recognition errors caused by semantic conflicts between modalities in the CH-SIMS dataset for EMT-DLFR. "Text Prediction" refers to results obtained using only the text modality, whereas "Multimodal Prediction" refers to results obtained using all three modalities. Note: In the CH-SIMS dataset, the annotated emotional polarity ranges from –1 to 1, with smaller values indicating more negative emotions.
• Figure 2: The overall framework of the CMC model. CMC mainly comprises three core modules: the Pseudo Label Generation Module (PLGM), the Parameter-free Fusion Module (PFM), and the Multimodal Consensus Router (MCR). In the unimodal pretraining stage, the PLGM generates pseudo unimodal labels to enable self-supervised pretraining of unimodal models. During the multimodal finetuning stage, the hidden representations of each modality are fused through the PFM, after which the MCR assigns weights to each modality. Finally, these weights are applied to perform weighted fusion of the unimodal model predictions, producing the final emotion recognition results.
• Figure 3: The overall framework of the CMC-variant model. During the multimodal finetuning stage, the parameters of the Encoder $f_k$ and Projector $p_k$ for all modalities are frozen, the PFM is removed, and the training is conducted by optimizing only the MCR component in a single-task setting.
• Figure 4: Performance comparison between CMC and CMC-GT on various test sets of CH-SIMS. "CMC-GT" denotes that ground truth unimodal labels are employed during unimodal pretraining.
• Figure 5: Performance comparison between CMC and CMC-GT on various test sets of CH-SIMS v2. "CMC-GT" denotes that ground truth unimodal labels are employed during unimodal pretraining.