Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models

Hong Jia, Weibin Li, Jingyao Wu, Xiaofeng Yu, Yan Gao, Jintao Cheng, Xiaoyu Tang, Feng Xia, Ting Dang

TL;DR

This work tackles the challenge of recognizing ambiguous emotions in speech by leveraging audio-language models (ALMs) and test-time scaling (TTS). It introduces the first TTS-enabled benchmark for ambiguous emotion recognition, evaluating eight ALMs across three datasets and comparing five inference-time strategies to produce robust emotion distributions rather than single labels. The study provides detailed analyses of how model capacity, TTS, and emotional ambiguity interact, highlighting that weighted aggregation methods (e.g., Weighted Best-of-N, Weighted ALM verifier) consistently improve distributional accuracy and uncertainty handling. The findings offer a foundation for developing context-aware, emotionally intelligent speech systems, while outlining practical directions for improving data diversity, annotation formats, and emotion-specific inference techniques.

Abstract

Emotion recognition from human speech is a critical enabler for socially aware conversational AI. However, while most prior work frames emotion recognition as a categorical classification problem, real-world affective states are often ambiguous, overlapping, and context-dependent, posing significant challenges for both annotation and automatic modeling. Recent large-scale audio language models (ALMs) offer new opportunities for nuanced affective reasoning without explicit emotion supervision, but their capacity to handle ambiguous emotions remains underexplored. At the same time, advances in inference-time techniques such as test-time scaling (TTS) have shown promise for improving generalization and adaptability in hard NLP tasks, but their relevance to affective computing is still largely unknown. In this work, we introduce the first benchmark for ambiguous emotion recognition in speech with ALMs under test-time scaling. Our evaluation systematically compares eight state-of-the-art ALMs and five TTS strategies across three prominent speech emotion datasets. We further provide an in-depth analysis of the interaction between model capacity, TTS, and affective ambiguity, offering new insights into the computational and representational challenges of ambiguous emotion understanding. Our benchmark establishes a foundation for developing more robust, context-aware, and emotionally intelligent speech-based AI systems, and highlights key future directions for bridging the gap between model assumptions and the complexity of real-world human emotion.

Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models

TL;DR

This work tackles the challenge of recognizing ambiguous emotions in speech by leveraging audio-language models (ALMs) and test-time scaling (TTS). It introduces the first TTS-enabled benchmark for ambiguous emotion recognition, evaluating eight ALMs across three datasets and comparing five inference-time strategies to produce robust emotion distributions rather than single labels. The study provides detailed analyses of how model capacity, TTS, and emotional ambiguity interact, highlighting that weighted aggregation methods (e.g., Weighted Best-of-N, Weighted ALM verifier) consistently improve distributional accuracy and uncertainty handling. The findings offer a foundation for developing context-aware, emotionally intelligent speech systems, while outlining practical directions for improving data diversity, annotation formats, and emotion-specific inference techniques.

Abstract

Emotion recognition from human speech is a critical enabler for socially aware conversational AI. However, while most prior work frames emotion recognition as a categorical classification problem, real-world affective states are often ambiguous, overlapping, and context-dependent, posing significant challenges for both annotation and automatic modeling. Recent large-scale audio language models (ALMs) offer new opportunities for nuanced affective reasoning without explicit emotion supervision, but their capacity to handle ambiguous emotions remains underexplored. At the same time, advances in inference-time techniques such as test-time scaling (TTS) have shown promise for improving generalization and adaptability in hard NLP tasks, but their relevance to affective computing is still largely unknown. In this work, we introduce the first benchmark for ambiguous emotion recognition in speech with ALMs under test-time scaling. Our evaluation systematically compares eight state-of-the-art ALMs and five TTS strategies across three prominent speech emotion datasets. We further provide an in-depth analysis of the interaction between model capacity, TTS, and affective ambiguity, offering new insights into the computational and representational challenges of ambiguous emotion understanding. Our benchmark establishes a foundation for developing more robust, context-aware, and emotionally intelligent speech-based AI systems, and highlights key future directions for bridging the gap between model assumptions and the complexity of real-world human emotion.
Paper Structure (34 sections, 3 equations, 4 figures, 8 tables)

This paper contains 34 sections, 3 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Speech Emotion Recognition using LLMs.
  4. Ambiguous Emotion Recognition.
  5. Test-Time Scaling.
  6. Problem Definition
  7. Methods
  8. Overview
  9. TTS methods
  10. CoT prompting
  11. Beam search with Best of N
  12. ALM verifier
  13. Experimental Setups
  14. Dataset.
  15. Implementation details.
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Problem definition of ambiguous emotion recognition using ALMs.
  • Figure 1: Performance of ALMs on CREMA-D dataset across different entropy bins. This analysis presents the median to mitigate the influence of occasional outliers on the overall model performance evaluation.
  • Figure 2: Test-time scaling strategies for ambiguous emotion recognition.
  • Figure 3: TTS performance with varying levels of ambiguity from low to high, quantified by entropy on CREMA-D.