Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Ambiguity: Augmenting Human Annotation in Speech Emotion Recognition with Audio-Language Models

Wenda Zhang, Hongyu Jin, Siyi Wang, Zhiqiang Wei, Ting Dang

TL;DR

This work tackles the challenge of ambiguous emotions in speech emotion recognition (SER) by adopting distributional labels (Ambiguity Emotion Recognition, AER) and addressing ground-truth sparsity through synthetic annotations generated by Audio-Language Models (ALMs). It introduces a three-component framework: Synthetic Perceptual Proxies to create diverse synthetic annotations, DiME-Aug to perform distribution-aware multimodal augmentation, and ALM Fine-tuning to train on augmented emotion distributions. Experiments on IEMOCAP and MSP-Podcast show that synthetic annotations can closely approximate human emotion distributions with about 6–10 synthetic samples per utterance and that combining synthetic with human annotations improves distributional accuracy in low-ambiguity regions, though benefits decline for highly ambiguous emotions. The results highlight the potential of ALMs to mitigate annotation bottlenecks in AER while underscoring the need for more advanced prompting and generation strategies to robustly handle high-ambiguity cases, with implications for HCI and mental health applications.

Abstract

Speech Emotion Recognition models typically use single categorical labels, overlooking the inherent ambiguity of human emotions. Ambiguous Emotion Recognition addresses this by representing emotions as probability distributions, but progress is limited by unreliable ground-truth distributions inferred from sparse human annotations. This paper explores whether Large Audio-Language Models (ALMs) can mitigate the annotation bottleneck by generating high-quality synthetic annotations. We introduce a framework leveraging ALMs to create Synthetic Perceptual Proxies, augmenting human annotations to improve ground-truth distribution reliability. We validate these proxies through statistical analysis of their alignment with human distributions and evaluate their impact by fine-tuning ALMs with the augmented emotion distributions. Furthermore, to address class imbalance and enable unbiased evaluation, we propose DiME-Aug, a Distribution-aware Multimodal Emotion Augmentation strategy. Experiments on IEMOCAP and MSP-Podcast show that synthetic annotations enhance emotion distribution, especially in low-ambiguity regions where annotation agreement is high. However, benefits diminish for highly ambiguous emotions with greater human disagreement. This work provides the first evidence that ALMs could address annotation scarcity in ambiguous emotion recognition, but highlights the need for more advanced prompting or generation strategies to handle highly ambiguous cases.

Scaling Ambiguity: Augmenting Human Annotation in Speech Emotion Recognition with Audio-Language Models

TL;DR

This work tackles the challenge of ambiguous emotions in speech emotion recognition (SER) by adopting distributional labels (Ambiguity Emotion Recognition, AER) and addressing ground-truth sparsity through synthetic annotations generated by Audio-Language Models (ALMs). It introduces a three-component framework: Synthetic Perceptual Proxies to create diverse synthetic annotations, DiME-Aug to perform distribution-aware multimodal augmentation, and ALM Fine-tuning to train on augmented emotion distributions. Experiments on IEMOCAP and MSP-Podcast show that synthetic annotations can closely approximate human emotion distributions with about 6–10 synthetic samples per utterance and that combining synthetic with human annotations improves distributional accuracy in low-ambiguity regions, though benefits decline for highly ambiguous emotions. The results highlight the potential of ALMs to mitigate annotation bottlenecks in AER while underscoring the need for more advanced prompting and generation strategies to robustly handle high-ambiguity cases, with implications for HCI and mental health applications.

Abstract

Speech Emotion Recognition models typically use single categorical labels, overlooking the inherent ambiguity of human emotions. Ambiguous Emotion Recognition addresses this by representing emotions as probability distributions, but progress is limited by unreliable ground-truth distributions inferred from sparse human annotations. This paper explores whether Large Audio-Language Models (ALMs) can mitigate the annotation bottleneck by generating high-quality synthetic annotations. We introduce a framework leveraging ALMs to create Synthetic Perceptual Proxies, augmenting human annotations to improve ground-truth distribution reliability. We validate these proxies through statistical analysis of their alignment with human distributions and evaluate their impact by fine-tuning ALMs with the augmented emotion distributions. Furthermore, to address class imbalance and enable unbiased evaluation, we propose DiME-Aug, a Distribution-aware Multimodal Emotion Augmentation strategy. Experiments on IEMOCAP and MSP-Podcast show that synthetic annotations enhance emotion distribution, especially in low-ambiguity regions where annotation agreement is high. However, benefits diminish for highly ambiguous emotions with greater human disagreement. This work provides the first evidence that ALMs could address annotation scarcity in ambiguous emotion recognition, but highlights the need for more advanced prompting or generation strategies to handle highly ambiguous cases.
Paper Structure (15 sections, 2 equations, 3 figures, 3 tables)

This paper contains 15 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Synthetic Perceptual Proxies
  5. DiME-Aug (Distribution-aware Multimodal Emotion Augmentation)
  6. Model Fine-tuning
  7. Experimental Setup
  8. Datasets
  9. Implementation Details
  10. Evaluations
  11. Results
  12. Similarity between Human and Synthetic Annotations
  13. Impact of Annotation Sources on AER
  14. Effectiveness of DiME-Aug
  15. Conclusion

Figures (3)

  • Figure 1: Overview of the proposed framework. The pipeline consists of three modules: (1) Synthetic Perceptual Proxies, which leverage ALMs to augment human annotations; (2) DiME-Aug, a distribution-aware multimodal augmentation to address class imbalance; and (3) Audio-Language Model (ALM) Fine-tuning.
  • Figure 2: JS Divergence (lower is better) vs. number of annotations on (a) IEMOCAP and (b) MSP-Podcast datasets. The dashed lines show fitted curves and saturation points.
  • Figure 3: JS divergence (lower is better) vs. ambiguity level on (a) IEMOCAP and (b) MSP-Podcast datasets. Samples are divided into three groups based on the human annotation's Shannon Entropy: Low, Medium, and High.