Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework

TL;DR

Multimodal emotion recognition systems falter when modalities are missing, limiting real-world applicability. The authors introduce CM-ARR, a Cross-Modal Alignment, Reconstruction, and Refinement framework that sequentially aligns modality distributions with unsupervised distribution-based contrastive learning, reconstructs missing modalities via normalizing flows, and refines emotional content through supervised point-based contrastive learning. Key innovations include Gaussian distribution modeling of modality representations, a 2-Wasserstein distance-based alignment objective, and a flow-based reconstruction pipeline that infers missing data from available modalities, followed by cross-modal fusion for classification; performance is demonstrated on IEMOCAP and MSP-IMPROV with improvements over SOTA when modalities are missing or complete. On average across six missing-modality scenarios, CM-ARR achieves absolute gains of $2.11\%$ in WAR and $2.12\%$ in UAR on IEMOCAP, and $1.71\%$ in WAR and $1.96\%$ in UAR on MSP-IMPROV, confirming robust efficacy in incomplete-data settings. The approach advances practical MMER by explicitly modeling semantic uncertainty, bridging cross-modal distribution gaps, and enabling reliable reconstruction and refinement of emotional content.

Abstract

Multimodal emotion recognition systems rely heavily on the full availability of modalities, suffering significant performance declines when modal data is incomplete. To tackle this issue, we present the Cross-Modal Alignment, Reconstruction, and Refinement (CM-ARR) framework, an innovative approach that sequentially engages in cross-modal alignment, reconstruction, and refinement phases to handle missing modalities and enhance emotion recognition. This framework utilizes unsupervised distribution-based contrastive learning to align heterogeneous modal distributions, reducing discrepancies and modeling semantic uncertainty effectively. The reconstruction phase applies normalizing flow models to transform these aligned distributions and recover missing modalities. The refinement phase employs supervised point-based contrastive learning to disrupt semantic correlations and accentuate emotional traits, thereby enriching the affective content of the reconstructed representations. Extensive experiments on the IEMOCAP and MSP-IMPROV datasets confirm the superior performance of CM-ARR under conditions of both missing and complete modalities. Notably, averaged across six scenarios of missing modalities, CM-ARR achieves absolute improvements of 2.11% in WAR and 2.12% in UAR on the IEMOCAP dataset, and 1.71% and 1.96% in WAR and UAR, respectively, on the MSP-IMPROV dataset.

Figures (5)

• Figure 1: An example of missing modalities: when the speech modality is missing, emotion recognition is guided by the text and video modalities, leading to incorrect predictions. The ground truth is "angry".
• Figure 2: The framework of CM-ARR consists of three phases: the alignment phase employs unsupervised distribution-based contrastive learning to semantically align the video, speech, and text modalities (see UMC in Fig. \ref{['fig:framework3']}); the reconstruction phase applies normalizing flow models to each modality; the refinement phase utilizes supervised point-based contrastive learning to accentuate emotional traits. The red arrows denote the inference process assuming the text modality is missing.
• Figure 3: The overall structure of the proposed UMC, where $g(\cdot)$ denotes the gelu function, $LN$ signifies the LayerNorm operation, and $MLP$ indicates the feed forward layer.
• Figure 4: The effect of weights $\alpha, \beta$, and $\lambda$ on performance.
• Figure 5: Visualization of the representations from different methods on the IEMOCAP corpus test set. Light blue represents speech reconstruction representations, while light red and light green depicts text and video reconstruction representations, respectively, with their corresponding darker shades indicating ground truth.