Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance

Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, Micha Elsner

TL;DR

The paper tackles whether audio-language models genuinely interpret speech by listening to acoustic cues or primarily rely on lexical content. It introduces LISTEN, a diagnostic benchmark with four conditions (Neutral-Text, Emotion-Matched, Emotion-Mismatched, Paralinguistic) that manipulate lexical–acoustic alignment across Text-only, Audio-only, and Text+Audio modalities, enabling zero-shot evaluation of six LALMs. Across results, models exhibit strong lexical dominance: text cues often drive predictions, while acoustic cues produce limited improvement, with paralinguistic signals achieving near-chance performance. The findings reveal that current LALMs tend to transcribe emotion from transcripts, not fully leverage speech signals, and LISTEN provides a principled framework to assess and guide future improvements in listening-based emotion understanding. The work suggests integrating robust prosodic grounding and acoustic-signal modeling, inspired by SER techniques, to enhance genuine multimodal emotion processing in LALMs.

Abstract

Understanding emotion from speech requires sensitivity to both lexical and acoustic cues. However, it remains unclear whether large audio language models (LALMs) genuinely process acoustic information or rely primarily on lexical content. We present LISTEN (Lexical vs. Acoustic Speech Test for Emotion in Narratives), a controlled benchmark designed to disentangle lexical reliance from acoustic sensitivity in emotion understanding. Across evaluations of six state-of-the-art LALMs, we observe a consistent lexical dominance. Models predict "neutral" when lexical cues are neutral or absent, show limited gains under cue alignment, and fail to classify distinct emotions under cue conflict. In paralinguistic settings, performance approaches chance. These results indicate that current LALMs largely "transcribe" rather than "listen," relying heavily on lexical semantics while underutilizing acoustic cues. LISTEN offers a principled framework for assessing emotion understanding in multimodal models.

Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance

TL;DR

The paper tackles whether audio-language models genuinely interpret speech by listening to acoustic cues or primarily rely on lexical content. It introduces LISTEN, a diagnostic benchmark with four conditions (Neutral-Text, Emotion-Matched, Emotion-Mismatched, Paralinguistic) that manipulate lexical–acoustic alignment across Text-only, Audio-only, and Text+Audio modalities, enabling zero-shot evaluation of six LALMs. Across results, models exhibit strong lexical dominance: text cues often drive predictions, while acoustic cues produce limited improvement, with paralinguistic signals achieving near-chance performance. The findings reveal that current LALMs tend to transcribe emotion from transcripts, not fully leverage speech signals, and LISTEN provides a principled framework to assess and guide future improvements in listening-based emotion understanding. The work suggests integrating robust prosodic grounding and acoustic-signal modeling, inspired by SER techniques, to enhance genuine multimodal emotion processing in LALMs.

Abstract

Understanding emotion from speech requires sensitivity to both lexical and acoustic cues. However, it remains unclear whether large audio language models (LALMs) genuinely process acoustic information or rely primarily on lexical content. We present LISTEN (Lexical vs. Acoustic Speech Test for Emotion in Narratives), a controlled benchmark designed to disentangle lexical reliance from acoustic sensitivity in emotion understanding. Across evaluations of six state-of-the-art LALMs, we observe a consistent lexical dominance. Models predict "neutral" when lexical cues are neutral or absent, show limited gains under cue alignment, and fail to classify distinct emotions under cue conflict. In paralinguistic settings, performance approaches chance. These results indicate that current LALMs largely "transcribe" rather than "listen," relying heavily on lexical semantics while underutilizing acoustic cues. LISTEN offers a principled framework for assessing emotion understanding in multimodal models.

Paper Structure

This paper contains 48 sections, 1 equation, 21 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Related work
  4. Audio Benchmarks
  5. Large Audio Language Models
  6. Speech Emotion Recognition
  7. The LISTEN Framework
  8. Data Construction
  9. Stage 1: Condition Design.
  10. Stage 2: Dataset Mapping.
  11. Stage 3: Question Generation.
  12. LISTEN Statistics
  13. Experiments
  14. Models
  15. Evaluation Protocol
  16. ...and 33 more sections

Figures (21)

  • Figure 1: Examples from the LISTEN benchmark.
  • Figure 2: Model accuracy across three LISTEN conditions (Neutral-Text, Emotion-Matched, Emotion-Mismatched) under text-only, audio-only, and text+audio modalities. Dashed lines indicate prediction-marginal baselines.
  • Figure 3: Confusion matrices showing Gemini 2.5 Pro's emotion recognition performance across three experimental conditions in the LISTEN benchmark. Row-normalized matrices display prediction distributions for each true emotion class across: (1) Neutral Text, (2) Emotion Matched, and (3) Emotion Mismatched conditions, each tested with text-only, audio-only, and audio+text modalities.
  • Figure 4: Confusion matrices showing Gemini 2.5 Pro's emotion recognition performance in paralinguistic condition
  • Figure 5: Ground-truth label distributions for Neutral-Text, Emotion-Matched, and Paralinguistic conditions. Each subplot shows counts for the labels present in that dataset.
  • ...and 16 more figures