TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognition

TL;DR

Inter-modal conflicts in multimodal emotion recognition challenge effective fusion when different modalities express divergent cues. TiCAL addresses this by generating unimodal pseudo labels via a high-confidence anchor samples list, embedding features in a hyperbolic space to reflect hierarchical emotion structure, and employing a typicality-based consistency metric to drive a three-stage, stage-wise fusion. The approach yields reliable per-sample consistency estimates and state-of-the-art results on MOSI, MOSEI, DFEW, and MER2023, including about a 2.6% gain over prior best on MOSI. Overall, TiCAL offers a robust, interpretable solution to conflict-aware MER by aligning unimodal tendencies with cross-modal fusion through human-like stage-wise perception.

Abstract

Multimodal Emotion Recognition (MER) aims to accurately identify human emotional states by integrating heterogeneous modalities such as visual, auditory, and textual data. Existing approaches predominantly rely on unified emotion labels to supervise model training, often overlooking a critical challenge: inter-modal emotion conflicts, wherein different modalities within the same sample may express divergent emotional tendencies. In this work, we address this overlooked issue by proposing a novel framework, Typicality-based Consistent-aware Multimodal Emotion Recognition (TiCAL), inspired by the stage-wise nature of human emotion perception. TiCAL dynamically assesses the consistency of each training sample by leveraging pseudo unimodal emotion labels alongside a typicality estimation. To further enhance emotion representation, we embed features in a hyperbolic space, enabling the capture of fine-grained distinctions among emotional categories. By incorporating consistency estimates into the learning process, our method improves model performance, particularly on samples exhibiting high modality inconsistency. Extensive experiments on benchmark datasets, e.g, CMU-MOSEI and MER2023, validate the effectiveness of TiCAL in mitigating inter-modal emotional conflicts and enhancing overall recognition accuracy, e.g., with about 2.6% improvements over the state-of-the-art DMD.

Figures (3)

• Figure 1: (a) Example of inter-modal emotional conflict where modalities show divergent affective cues. (b) Comparison of our paradigm with prior work: we model modality-specific emotional tendencies and enforce consistency-aware dynamic fusion, inspired by human perception. $\tau_m$ and $\kappa$ represent typicality and consistency calculated using our method, respectively.
• Figure 2: The overview of our proposed TiCAL. TiCAL comprises Reliable Consistency Estimation and Human-like Stage-wise Perception. Specifically, we generate unimodal pseudo labels $y_m^*$ using a high-confidence anchor samples list (HASL) and regularize them in the hyperbolic space (In § 3.1). Then we estimate the reliable inter-modal consistency $\kappa$ based on typicality $\tau_m$ and unimodal pseudo labels $y_m^*$ (In § 3.2). Inspired by human emotion perception, we design a three-stage prediction structure—EP,CI, and AC—and conduct dynamic typicality-based consistency-aware optimization (In § 3.3).
• Figure 3: The performance of our TiCAL method with different confidence threshold $\theta$ in HASL initialization. The left side is on MOSI and the right side is on MOSEI.

Theorems & Definitions (4)

• Theorem 1: Poincaré Ball Model
• Definition 1: Tree Node Pair Distance
• Definition 2: Unimodal typicality
• Definition 3: Inter-modal Consistency