Can Large Audio Language Models Understand Audio Well? Speech, Scene and Events Understanding Benchmark for LALMs

TL;DR

SSEU-Bench is introduced, the first versatile audio understanding benchmark that explicitly accounts for energy differences between speech and non-speech audio, and Chain-of-Thought is introduced, which effectively improves LALMs' joint audio understanding performance by decomposing complex tasks into simpler reasoning steps.

Abstract

Recently, Large Audio Language Models (LALMs) have progressed rapidly, demonstrating their strong efficacy in universal audio understanding through cross-modal integration. To evaluate LALMs' audio understanding performance, researchers have proposed different benchmarks. However, key aspects for real-world interactions are underexplored in existing benchmarks, i.e., audio signals typically contain both speech and non-speech components, and energy levels of these components can vary significantly across different scenarios. Moreover, most benchmarks do not consider the joint understanding of speech, scene, and events within the same audio clip. In this work, we introduce SSEU-Bench, the first versatile audio understanding benchmark that explicitly accounts for energy differences between speech and non-speech audio, with both independent and joint understanding settings for speech, scene, and events. Furthermore, we demonstrate that some LALMs tend to underperform on certain tasks in a joint understanding setting. To address this issue, we introduce Chain-of-Thought, which effectively improves LALMs' joint audio understanding performance by decomposing complex tasks into simpler reasoning steps.

Figures (3)

• Figure 1: Overview of the proposed Speech, Scene and Events Understanding Benchmark (SSEU-Bench).
• Figure 2: Overview of the Scene and Events Understanding Evaluation Methods in SSEU-Bench.
• Figure 3: Performance of Step-Audio 2 Mini with different understanding strategies on SSEU-Bench.