AmbER$^2$: Dual Ambiguity-Aware Emotion Recognition Applied to Speech and Text

TL;DR

AmbER2 addresses dual ambiguity in emotion recognition by modeling both inter-rater label distributions and modality-conflict signals within a teacher–student framework. It introduces a distributional supervision objective with two losses: $L_{RAI}$, which aligns the student with the ground-truth rater distribution via Jensen–Shannon divergence $JS(\cdot \parallel \cdot)$, and $L_{MAI}$, which adaptively weights modality-specific experts by their agreement with the distribution using weights $u_m$ derived from $D_m = JS(p_m \parallel y)$. Evaluations on IEMOCAP and MSP-Podcast show improved distributional fidelity (lower $JS$, higher $BC$, higher $R^2$) and competitive classification metrics, outperforming baselines and several state-of-the-art models. The results demonstrate the value of explicitly modeling dual ambiguity, especially for highly ambiguous samples, with a framework generalizable across modalities and languages.

Abstract

Emotion recognition is inherently ambiguous, with uncertainty arising both from rater disagreement and from discrepancies across modalities such as speech and text. There is growing interest in modeling rater ambiguity using label distributions. However, modality ambiguity remains underexplored, and multimodal approaches often rely on simple feature fusion without explicitly addressing conflicts between modalities. In this work, we propose AmbER$^2$, a dual ambiguity-aware framework that simultaneously models rater-level and modality-level ambiguity through a teacher-student architecture with a distribution-wise training objective. Evaluations on IEMOCAP and MSP-Podcast show that AmbER$^2$ consistently improves distributional fidelity over conventional cross-entropy baselines and achieves performance competitive with, or superior to, recent state-of-the-art systems. For example, on IEMOCAP, AmbER$^2$ achieves relative improvements of 20.3% on Bhattacharyya coefficient (0.83 vs. 0.69), 13.6% on R$^2$ (0.67 vs. 0.59), 3.8% on accuracy (0.683 vs. 0.658), and 4.5% on F1 (0.675 vs. 0.646). Further analysis across ambiguity levels shows that explicitly modeling ambiguity is particularly beneficial for highly uncertain samples. These findings highlight the importance of jointly addressing rater and modality ambiguity when building robust emotion recognition systems.

Figures (2)

• Figure 1: An graphical illustration of the proposed AmbER2 framework when Fusion Head is considered as Student.
• Figure 2: Performance comparison of AmbER2 and baseline at different ambiguity levels.