Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios

Yibo Zhang, Liang Lin, Kaiwen Luo, Shilinlu Yan, Jin Wang, Yaoqi Guo, Yitian Chen, Yalan Qin, Zhenhong Zhou, Kun Wang, Li Sun

TL;DR

RSA-Bench quantifies ALLM robustness in authentic acoustic ecologies by constructing over 100k samples across four real-world environments and 17 interference configurations, enabling evaluation from ASR to open-ended reasoning. The framework combines a multi-source superposition pipeline with RMS-energy alignment to simulate realistic noise, paired with a diverse model roster and LL M-based semantic evaluation. Key findings reveal a pronounced Perception-Cognition Gap, strong resilience of some perception tasks (e.g., GR) to interference, and severe erosion of semantic understanding and reasoning under acoustic stress. A denoising paradox emerges: conventional speech enhancement often degrades ALLM performance due to artifacts, underscoring the need for noise-aware learning and robust, intrinsic models rather than post-hoc patching.

Abstract

While Audio Large Models (ALMs) have achieved remarkable proficiency, their robustness remains brittle in real-world deployment. Existing evaluations largely rely on synthetic Gaussian noise or simplistic single-source interference, failing to capture the intricate, multi-layered acoustic dynamics -- or ``Acoustic Ecology'' -- that characterize authentic physical environments. To bridge this ecological gap, we introduce \textbf{RSA-Bench}, a comprehensive robustness benchmark designed to stress-test ALLMs through high-fidelity auditory scene simulations. Unlike traditional methods, we construct evaluation samples by naturally superimposing diverse environmental soundscapes -- spanning \textit{Pasture}, \textit{Extreme Weather}, \textit{Classroom}, and \textit{Outdoors} -- onto clean speech signals across a spectrum of interference intensities. By evaluating models on six core tasks ranging from fundamental perception to complex reasoning, our study unveils three macro-level insights: \textbf{(I) The Perception-Cognition Gap:} Models maintain relative resilience in low-level recognition but suffer a \textbf{functional collapse} in high-order reasoning tasks under stress; \textbf{(II) Scenario Sensitivity:} ``Vocal-like'' interference (e.g., background laughter) proves significantly more destructive than mechanical noise, challenging the model's auditory attention mechanisms; and \textbf{(III) The Denoising Paradox:} Standard speech enhancement often exacerbates performance degradation, as ALLMs prove highly sensitive to the semantic distortions introduced by denoising artifacts.

RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios

TL;DR

RSA-Bench quantifies ALLM robustness in authentic acoustic ecologies by constructing over 100k samples across four real-world environments and 17 interference configurations, enabling evaluation from ASR to open-ended reasoning. The framework combines a multi-source superposition pipeline with RMS-energy alignment to simulate realistic noise, paired with a diverse model roster and LL M-based semantic evaluation. Key findings reveal a pronounced Perception-Cognition Gap, strong resilience of some perception tasks (e.g., GR) to interference, and severe erosion of semantic understanding and reasoning under acoustic stress. A denoising paradox emerges: conventional speech enhancement often degrades ALLM performance due to artifacts, underscoring the need for noise-aware learning and robust, intrinsic models rather than post-hoc patching.

Abstract

While Audio Large Models (ALMs) have achieved remarkable proficiency, their robustness remains brittle in real-world deployment. Existing evaluations largely rely on synthetic Gaussian noise or simplistic single-source interference, failing to capture the intricate, multi-layered acoustic dynamics -- or ``Acoustic Ecology'' -- that characterize authentic physical environments. To bridge this ecological gap, we introduce \textbf{RSA-Bench}, a comprehensive robustness benchmark designed to stress-test ALLMs through high-fidelity auditory scene simulations. Unlike traditional methods, we construct evaluation samples by naturally superimposing diverse environmental soundscapes -- spanning \textit{Pasture}, \textit{Extreme Weather}, \textit{Classroom}, and \textit{Outdoors} -- onto clean speech signals across a spectrum of interference intensities. By evaluating models on six core tasks ranging from fundamental perception to complex reasoning, our study unveils three macro-level insights: \textbf{(I) The Perception-Cognition Gap:} Models maintain relative resilience in low-level recognition but suffer a \textbf{functional collapse} in high-order reasoning tasks under stress; \textbf{(II) Scenario Sensitivity:} ``Vocal-like'' interference (e.g., background laughter) proves significantly more destructive than mechanical noise, challenging the model's auditory attention mechanisms; and \textbf{(III) The Denoising Paradox:} Standard speech enhancement often exacerbates performance degradation, as ALLMs prove highly sensitive to the semantic distortions introduced by denoising artifacts.
Paper Structure (33 sections, 5 equations, 3 figures, 12 tables)

This paper contains 33 sections, 5 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RSA-Bench
  4. Data Construction
  5. Step 1. Source Collection.
  6. Step 2. Temporal Alignment.
  7. Step 3. RMS-based Energy Alignment.
  8. Step 4. Superposition and Dynamic Constraint.
  9. Task Taxonomy and Definitions
  10. Perception & Paralinguistics.
  11. Cognitive Reasoning.
  12. Experiments
  13. Main Results
  14. Impact of Real-world Interference (RQ1)
  15. Obs 1: Differentiation within Perception Tasks.
  16. ...and 18 more sections

Figures (3)

  • Figure 1: A framework of our RSA-Benchmark for evaluating Audio-LLM robustness across six different tasks.
  • Figure 2: Overview of the RSA-Bench data composition, which covers 6 tasks and totals over 100,000 samples.
  • Figure 3: Evaluation case study: In an audio gender recognition task, the model misidentifies a male speaker as female and includes irrelevant details.