Estimating the Uncertainty in Emotion Class Labels with Utterance-Specific Dirichlet Priors

TL;DR

This work tackles inherent uncertainty in emotion labelling by treating utterance annotations as samples from an utterance-specific Dirichlet prior. It introduces Dirichlet Prior Network (DPN) training, optionally combined with soft-label KL loss, to model per-utterance emotion distributions and uncertainty. The approach achieves state-of-the-art emotion classification on IEMOCAP while enabling uncertainty estimation via metrics like AUPR for detecting utterances without majority labels, and generalizes to MSP-Podcast. The combination of a two-branch architecture (TSB/TAB) with uncertainty-aware training yields robust performance and practical insight for handling ambiguous emotional content in real-world data.

Abstract

Emotion recognition is a key attribute for artificial intelligence systems that need to naturally interact with humans. However, the task definition is still an open problem due to the inherent ambiguity of emotions. In this paper, a novel Bayesian training loss based on per-utterance Dirichlet prior distributions is proposed for verbal emotion recognition, which models the uncertainty in one-hot labels created when human annotators assign the same utterance to different emotion classes. An additional metric is used to evaluate the performance by detection test utterances with high labelling uncertainty. This removes a major limitation that emotion classification systems only consider utterances with labels where the majority of annotators agree on the emotion class. Furthermore, a frequentist approach is studied to leverage the continuous-valued "soft" labels obtained by averaging the one-hot labels. We propose a two-branch model structure for emotion classification on a per-utterance basis, which achieves state-of-the-art classification results on the widely used IEMOCAP dataset. Based on this, uncertainty estimation experiments were performed. The best performance in terms of the area under the precision-recall curve when detecting utterances with high uncertainty was achieved by interpolating the Bayesian training loss with the Kullback-Leibler divergence training loss for the soft labels. The generality of the proposed approach was verified using the MSP-Podcast dataset which yielded the same pattern of results.

Figures (7)

• Figure 1: Illustration of the DPN process. $\boldsymbol{\mu}$ is a categorical distribution over $K$ emotion classes which is sampled from the Dirichlet prior $\operatorname{Dir}(\boldsymbol{\mu}|{\boldsymbol{{\alpha}}})$.
• Figure 2: Proposed two-branch model structure.
• Figure 3: Error bars showing the standard deviation across 5-fold for uncertainty estimation experiments on IEMOCAP. The last two figures show the AUPR value for each fold where "Fold1" denotes the 1st fold that was trained on Session 2-5 and tested on Session 1, etc.
• Figure 4: Comparison of the "hard" system and the "soft" system in terms of KL divergence (a) and entropy (b) of three data groups in Session 5 of IEMOCAP.
• Figure 5: PR curves for the four systems using (a) Max.P and (b) Ent. as the uncertainty measures. The tests were performed on Session 5 of IEMOCAP.