CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges

TL;DR

CMDAR targets the gap in Chinese, multi-scene, dynamic audio reasoning by releasing a 3,000-question benchmark drawn from real audio clips across five reasoning categories and three question types. It introduces a three-part data pipeline (data collection from Chinese films, an audio-to-description generation process, and expert QA annotation) and three evaluation modes (CMDAR-main, CMDAR-open, CMDAR-multi) with tailored metrics. Across 26 models, CMDAR reveals substantial challenges, especially for open-ended and multi-audio tasks, and highlights a persistent gap between open-source and closed-source systems, as well as the perceptual versus reasoning limitations of current LALMs. The authors also provide analyses of cascaded caption models, robustness to noise, and instruction biases, offering concrete guidance for future Chinese audio-language model development and data-centric improvements.

Abstract

The ability to reason from audio, including speech, environmental sounds, and music, is essential for AI agents to interact effectively in real-world scenarios. Existing benchmarks mainly focus on static or single-scene settings and English audio data and do not fully capture scenarios where multiple speakers, unfolding events, and heterogeneous audio sources interact. To address these challenges, we introduce CMDAR, a Chinese benchmark for evaluating models on complex, multi-scene, and dynamically evolving audio reasoning tasks. CMDAR comprises 3,000 carefully curated question-answer pairs linked to diverse audio clips, covering five categories of complex reasoning and spanning three question types. We benchmark 26 state-of-the-art audio language models on CMDAR and observe that they exhibit limitations in complex reasoning tasks. In CMDAR-main, Qwen2.5-Omni achieves 76.67% accuracy, whereas GPT-4o Audio reaches 68.47%. However, GPT-4o Audio substantially outperforms Qwen2.5-Omni on the more challenging multiple-choice with multiple audios and open-ended tasks. And we provide detail analysis corresponding suggestions for the future development of large audio language models.

Figures (5)

• Figure 1: The differences between the existing benchmark and CMDAR. Tasks in CMDAR are essential skills for AI Agent.
• Figure 2: Examples from CMDAR showcase the diversity of complex dynamic reasoning tasks and also reveal the design of our newly proposed multiple-choice questions. These tasks not only demonstrate the depth of CMDAR in multidimensional capability assessment but also reflect its high standards of challenge.
• Figure 3: Data Construction Pipeline: It consists of three parts: Data Preparation, Audio Pipeline, and Data Quality Assurance, to ensure high quality and consistency of the data.
• Figure 4: Left: Comparison of different combinations with the same caption model or the same reasoning model in the cascaded models. Middle: When Gaussian noise is used to replace the audio on CMDAR-main to test four models, the performance of all models experience a significant drop. Right: Distribution of error types
• Figure 5: Distribution of all types of tasks included in the three benchmark tests illustrates the diversity and complexity of CMDAR tasks.