SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Models

TL;DR

SonicBench demonstrates that large audio language models struggle to perceive and reason about core physical audio attributes, despite strong semantic capabilities. By employing a psychophysically grounded benchmark of 12 attributes across five dimensions and two tasks (recognition and comparison), the study uncovers a bottleneck in alignment and decoding rather than perceptual encoding, as linear probes on frozen encoders reliably recover physical cues while end-to-end models falter. Humans consistently outperform models and exhibit a clear comparison advantage, which current architectures fail to replicate, suggesting a need for architectures or training strategies that support explicit relational reasoning over audio signals. The work provides a reproducible benchmark and toolbox for controlled stimulus generation, offering a concrete pathway to build more robust, physically grounded LALMs for real-world audio understanding and interaction.

Abstract

Large Audio Language Models (LALMs) excel at semantic and paralinguistic tasks, yet their ability to perceive the fundamental physical attributes of audio such as pitch, loudness, and spatial location remains under-explored. To bridge this gap, we introduce SonicBench, a psychophysically grounded benchmark that systematically evaluates 12 core physical attributes across five perceptual dimensions. Unlike previous datasets, SonicBench uses a controllable generation toolbox to construct stimuli for two complementary paradigms: recognition (absolute judgment) and comparison (relative judgment). This design allows us to probe not only sensory precision but also relational reasoning capabilities, a domain where humans typically exhibit greater proficiency. Our evaluation reveals a substantial deficiency in LALMs' foundational auditory understanding; most models perform near random guessing and, contrary to human patterns, fail to show the expected advantage on comparison tasks. Furthermore, explicit reasoning yields minimal gains. However, our linear probing analysis demonstrates crucially that frozen audio encoders do successfully capture these physical cues (accuracy at least 60%), suggesting that the primary bottleneck lies in the alignment and decoding stages, where models fail to leverage the sensory signals they have already captured.

Figures (16)

• Figure 1: Taxonomy of SonicBench acoustic attributes. Overview of the twelve perceptual attributes evaluated in SonicBench, organized into five dimensions: Spectral & Amplitude, Temporal, Spatial & Environment, Timbre, and Scene-Level. Each panel illustrates the auditory concept tested and the corresponding binary judgment required from models. This taxonomy defines the perceptual scope of SonicBench, systematically covering the full range of low-level sound features to high-level scene reasoning. Examples shown here are adapted for illustration purposes; original benchmark samples are provided in Appendix \ref{['appendix:task_instruction']}.
• Figure 2: A comprehensive pipeline for constructing SonicBench. The process includes: (1) Brain Storm collecting ideas through AI-assisted brainstorming and literature review; (2) Taxonomy & Task Design defining perceptual dimensions and formulating recognition and comparison tasks; (3) Data Preparation selecting and filtering sound sources to ensure diversity and balance; (4) Annotation generating QA templates and structured JSON metadata for both task types; and (5) Review & Double Check performing multi-round manual validation including quality control, significance testing, correction, and finalization to ensure benchmark reliability.
• Figure 3: Overall SonicBench accuracy across 36 systems. Bars show mean accuracy over 12 attributes and 2 tasks. Colors represent model categories. Dashed lines mark each family’s mean $\pm$ SE (standard error) and the gray line marks the random-guess baseline (0.5). OLMs achieve the highest overall accuracy, while many systems cluster near chance.
• Figure 4: A Case of reasoning cannot correcting perceptual failures. Both receive the same duration recognition prompt. The model in base mode directly answers wrong, while in think mode produces a logically coherent reasoning but fails to rectify upstream perceptual errors. A case exemplifies reasoning-induced perceptual errors in Figure \ref{['fig:case_study']}.
• Figure 5: Attribute-level overall accuracy distribution across model families. Violin plots show per-attribute accuracy for all models, with background colors denoting attribute dimensions and markers indicating family means. Velocity, Direction, and Distance consistently exhibit near-chance performance (no model exceeds 0.6 accuracy), indicating persistent difficulties in capturing spatial and motion-related cues.