Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation

TL;DR

This work tackles the challenge of representing multimodal signals for emotion recognition in conversation by explicitly disentangling unique, redundant, and synergistic information components via Partial Information Decomposition (PID). It introduces Divide and Refine (DnR), a two-phase, plug-in framework that first decomposes modality embeddings into $U$, $R$, and $S$ streams and then strengthens redundancy through redundancy-augmented, contrastive refinement, without altering backbone architectures. The approach yields consistent improvements across multiple MERC backbones on IEMOCAP and MELD and includes ablations, decomposition visualizations, and qualitative case studies to illustrate the PID-based dynamics. By promoting a principled separation of informational roles and robust cross-modal cues, DnR enhances both the accuracy and interpretability of MERC models, with potential to generalize to other multimodal tasks and scenarios with missing modalities.

Abstract

Multimodal emotion recognition in conversation (MERC) requires representations that effectively integrate signals from multiple modalities. These signals include modality-specific cues, information shared across modalities, and interactions that emerge only when modalities are combined. In information-theoretic terms, these correspond to \emph{unique}, \emph{redundant}, and \emph{synergistic} contributions. An ideal representation should leverage all three, yet achieving such balance remains challenging. Recent advances in contrastive learning and augmentation-based methods have made progress, but they often overlook the role of data preparation in preserving these components. In particular, applying augmentations directly to raw inputs or fused embeddings can blur the boundaries between modality-unique and cross-modal signals. To address this challenge, we propose a two-phase framework \emph{\textbf{D}ivide and \textbf{R}efine} (\textbf{DnR}). In the \textbf{Divide} phase, each modality is explicitly decomposed into uniqueness, pairwise redundancy, and synergy. In the \textbf{Refine} phase, tailored objectives enhance the informativeness of these components while maintaining their distinct roles. The refined representations are plug-and-play compatible with diverse multimodal pipelines. Extensive experiments on IEMOCAP and MELD demonstrate consistent improvements across multiple MERC backbones. These results highlight the effectiveness of explicitly dividing, refining, and recombining multimodal representations as a principled strategy for advancing emotion recognition. Our implementation is available at https://github.com/mattam301/DnR-WACV2026

Figures (4)

• Figure 1: Illustrative examples of modality interactions in MER. Left: verbal, acoustic, and visual cues consistently convey anger, reflecting redundancy across modalities. Right: text and visual cues redundantly suggest positivity, yet the unique acoustic cue (fast, low tone) provides critical evidence which, when integrated synergistically, reveals the true emotion of nervousness. These cases demonstrate that unique information, though often overshadowed by redundancy, plays a vital role in synergy and accurate emotion recognition.
• Figure 2: Overall Framework of DnR, with two consecutive phases: Divide and Refine. Divide phase is conducted first as all outputs of this phase are concatenated to make predict to classification task with ${L_{corr}}$ and ${L_{uncor}}$ as auxiliary for separating uniqueness, redundancy and synergy. All trainable parameters of this phase is frozen before Refine phase
• Figure 3: t-SNE visualization of the PID process embeddings at three training stages (epoch 0, 10, 100). Each subfigure shows the 2-D t-SNE projection; colors correspond to class/label if provided.
• Figure 4: Ablation study of DnR conducted on IEMOCAP dataset.