Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion Recognition

TL;DR

This work tackles multimodal multi-label emotion recognition by modeling aleatoric uncertainty in a latent emotion space. It introduces LDDU, combining a Transformer-based unimodal extractor, a contrastive disentangled representation in a Gaussian latent space, and an uncertainty-aware fusion with calibration to integrate semantic features with distributional uncertainty. The approach yields state-of-the-art results on CMU-MOSEI and M^3ED, with notable improvements on unaligned data and robust performance across metrics, validating the importance of uncertainty modeling in MMER. The framework advances affective computing by providing probabilistic emotion-space modeling, distribution-aware fusion, and calibrated uncertainty as core design principles, enabling more robust and interpretable multimodal emotion recognition.

Abstract

Multimodal multi-label emotion recognition (MMER) aims to identify the concurrent presence of multiple emotions in multimodal data. Existing studies primarily focus on improving fusion strategies and modeling modality-to-label dependencies. However, they often overlook the impact of \textbf{aleatoric uncertainty}, which is the inherent noise in the multimodal data and hinders the effectiveness of modality fusion by introducing ambiguity into feature representations. To address this issue and effectively model aleatoric uncertainty, this paper proposes Latent emotional Distribution Decomposition with Uncertainty perception (LDDU) framework from a novel perspective of latent emotional space probabilistic modeling. Specifically, we introduce a contrastive disentangled distribution mechanism within the emotion space to model the multimodal data, allowing for the extraction of semantic features and uncertainty. Furthermore, we design an uncertainty-aware fusion multimodal method that accounts for the dispersed distribution of uncertainty and integrates distribution information. Experimental results show that LDDU achieves state-of-the-art performance on the CMU-MOSEI and M$^3$ED datasets, highlighting the importance of uncertainty modeling in MMER. Code is available at https://github.com/201983290498/lddu\_mmer.git.

Figures (9)

• Figure 1: An illustration of aleatoric uncertainty in MMER task. When adopting Gaussian distribution modeling in latent emotion space, case two's semantic feature is more fuzzed with a larger variance due to cold speaking style than case one. Meanwhile, case one has stronger emotion intense located closer to the center of global distribution.
• Figure 2: The proposed LDDU framework consists of three components: (1) the transformer-base unimodal extractor (2) a contrastive learning-based emotion space decomposition module , and (3) an uncertainty-aware fusion and uncertainty calibration module.
• Figure 3: In the latent emotion space, we decouple emotion-related, modality-specific samples into separate distributions and use CL to group samples of the same category together while separating samples from different categories.
• Figure 5: The case of emotion recognition by multiple methods. Visual and acoustic modalities revealed a shift from sadness to anger, while the textual modality explicitly indicated anger-related expressions.
• Figure 6: Prompts of MLLMs.