Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models

Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Weisheng Xu, Ziteng Wang, Ruofan Liao, Yutong Zhang, Sichen Liu

Abstract

Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.

DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models

Abstract

Recent Audio Multimodal Large Language Models (Audio MLLMs) demonstrate impressive performance on speech benchmarks, yet it remains unclear whether these models genuinely process acoustic signals or rely on text-based semantic inference. To systematically study this question, we introduce DEAF (Diagnostic Evaluation of Acoustic Faithfulness), a benchmark of over 2,700 conflict stimuli spanning three acoustic dimensions: emotional prosody, background sounds, and speaker identity. Then, we design a controlled multi-level evaluation framework that progressively increases textual influence, ranging from semantic conflicts in the content to misleading prompts and their combination, allowing us to disentangle content-driven bias from prompt-induced sycophancy. We further introduce diagnostic metrics to quantify model reliance on textual cues over acoustic signals. Our evaluation of seven Audio MLLMs reveals a consistent pattern of text dominance: models are sensitive to acoustic variations, yet predictions are predominantly driven by textual inputs, revealing a gap between high performance on standard speech benchmarks and genuine acoustic understanding.
Paper Structure (49 sections, 4 equations, 6 figures, 9 tables)

This paper contains 49 sections, 4 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The DEAF Benchmark
  3. Three Conflict Types
  4. Emotion-Semantic Conflict (ESC).
  5. Background Sound-Semantic Conflict (BSC).
  6. Speaker Identity-Semantic Conflict (SIC).
  7. Question Design
  8. Three Levels of Textual Interference
  9. Level 1: Acoustic vs. Semantic Content.
  10. Level 2: Acoustic vs. Misleading Prompt.
  11. Level 3: Acoustic vs. Semantic + Prompt (Dual Interference).
  12. Prompt Templates.
  13. LLM-as-Judge for Open-ended Evaluation
  14. Evaluation Metrics
  15. Accuracy (Acc).
  16. ...and 34 more sections

Figures (6)

  • Figure 1: Illustration of the three acoustic--semantic conflict types in DEAF. ESC: the text expresses happiness while the vocal tone conveys depression. BSC: the text implies a quiet setting while the audio contains noisy traffic. SIC: the text implies an elderly female speaker while the voice belongs to a young boy. In each case, the correct answer requires following the audio signal, not the text.
  • Figure 2: Word clouds of the happy class under Explicit and Implicit semantic conditions in ESC.
  • Figure 3: Overview of the DEAF dataset construction pipeline.
  • Figure 4: Overview of the DEAF framework. Each conflict type is evaluated at three levels of increasing textual interference. Correct answers always require following acoustic evidence; trap answers follow textual cues.
  • Figure 5: Radar comparison of acoustic perception performance across conflict types.
  • ...and 1 more figures