Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

Hezhao Zhang, Huang-Cheng Chou, Shrikanth Narayanan, Thomas Hain

TL;DR

VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning, and introduces a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement.

Abstract

Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.

VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs

TL;DR

VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning, and introduces a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement.

Abstract

Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions.
Paper Structure (8 sections, 1 figure, 2 tables)

This paper contains 8 sections, 1 figure, 2 tables.

Table of Contents

  1. Soft-Label Evaluation
  2. Impacts of Prompts and Zero-Shot Instability
  3. Mitigating Instability via Prompt Ensemble
  4. Qwen2-Audio vs. Audio Flamingo 3
  5. Capturing Affective Ambiguity and Subjectivity
  6. Cross-Corpus Evaluation
  7. Conclusion, Limitation and Future Work
  8. Generative AI Use Disclosure

Figures (1)

  • Figure 1: Impact of zero-shot prompt complexity on performance and format adherence. The blue bars (left axis) denote the Average Macro-F1 across five evaluation corpora, while the red line (right axis) illustrates the Average Parse Failure Rate. As prompt constraints increase (from direct classification Direct to requesting intermediate acoustic captions +T), both Q2A and AF3 struggle to follow the structured text-output format, leading to a severe degradation in overall predictive performance. This zero-shot stochasticity necessitates our proposed distribution-aware prompt ensemble methodology.