Hearing is Believing? Evaluating and Analyzing Audio Language Model Sycophancy with SYAUDIO

TL;DR

SYAUDIO introduces the first benchmark dedicated to evaluating sycophancy in Audio Language Models, aggregating 4,319 audio questions across perception, reasoning, math, and ethics from MMAR, MMAU, GSM8K-Audio, and MMLU-Audio. It systematically perturbs prompts and audio conditions (noise and speaking rate) and measures Misleading Susceptibility Score ($MSS$) and Correction Receptiveness Score ($CRS$) to quantify alignment tendencies. The study finds that audio inputs amplify sycophancy relative to text, with strong sensitivity to prompt type and task, while background noise has limited impact and rate acts as a modest modulator. A supervised fine-tuning approach using chain-of-thought data reduces $MSS$ more consistently than it improves $CRS$, suggesting the need for correction-centered training objectives and conservative deployment in safety-critical contexts. Overall, SYAUDIO provides a benchmark-driven path toward safer, evidence-grounded ALMs and highlights directions for mitigating audio-specific sycophancy in real-world use cases.

Abstract

Audio Language Models (ALMs) have recently shown strong capabilities in unified reasoning over speech, sound, and natural language; yet they inherit behavioral issues observed in Large Language Models, including sycophancy--the tendency to agree with user assertions even when they contradict objective evidence. While sycophancy has been extensively studied in text and vision-language models, its manifestation in audio-conditioned reasoning remains largely unexplored, despite the need for ALMs to rely on auditory cues such as acoustic events, speaker characteristics, and speech rate. To address this gap, we introduce SYAUDIO, the first benchmark dedicated to evaluating sycophancy in ALMs, consisting of 4,319 audio questions spanning Audio Perception, Audio Reasoning, Audio Math, and Audio Ethics. Built upon established audio benchmarks and augmented with TTS-generated arithmetic and moral reasoning tasks, SYAUDIO enables systematic evaluation across multiple domains and sycophancy types with carefully verified data quality. Furthermore, we analyze audio-specific sycophancy under realistic conditions involving noise and rate, and demonstrate that supervised fine-tuning with chain-of-thought data is an effective mitigation strategy for reducing sycophantic behavior in ALMs.

Figures (11)

• Figure 1: Examples of four types of audio sycophancy tasks, where ALMs produce different answers in response to different user cues.
• Figure 2: Overview of the SYAUDIO pipeline. The figure shows how user cues interact with audio evidence in ALMs, leading to potential sycophantic behaviors. We build SYAUDIO from multiple audio task categories (perception, reasoning, math, and ethics) with TTS generation and quality control. We then run a multi-round protocol under diverse user cues and audio-specific conditions to compute MSS and CRS, and apply supervised fine-tuning to mitigate sycophancy with before–after behavioral analysis.
• Figure 3: Per-model average MSS (top row; lower is better) and CRS (bottom row; higher is better), aggregated over all sycophancy scenarios and reported separately for each dataset. While closed-source models achieve strong performance on specific datasets, the open-source Qwen2.5-Omni-7B exhibits comparable overall behavior across both MSS and CRS, indicating that its global sycophancy characteristics are on par with those of competitive closed-source audio language models.
• Figure 4: Average MSS under different bias feedback strengths (Strong, Medium, Low) across datasets and models. Overall, MSS tends to decrease as the bias strength weakens from Strong to Low, indicating reduced sycophantic behavior under milder feedback. However, several models and datasets exhibit non-monotonic trends, suggesting that the effect of bias strength is not strictly consistent and that bias feedback does not universally induce stronger sycophancy.
• Figure 5: Comparison of MSS and Robustness CRS between baseline inputs and TTS-generated audio. TTS inputs significantly increase MSS across all categories without degrading CRS.