Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Benchmark for Audio Reasoning Capabilities of Multimodal Large Language Models

Iwona Christop, Mateusz Czyżnikiewicz, Paweł Skórzewski, Łukasz Bondaruk, Jakub Kubiak, Marcin Lewandowski, Marek Kubis

TL;DR

The paper tackles the problem of evaluating multimodal LLMs on reasoning over complex audio signals by introducing Audio Reasoning Tasks (ART), a template-driven benchmark comprising $9$ tasks that require cross-modal audio understanding and inference. ART combines question, utterance, and sound prompts generated with voice cloning and synthetic speech, enabling end-to-end assessment without leaning on single-task modules. Through two evaluation paradigms (Yes/No and Descriptive) and multiple baselines, the study demonstrates that even strong Audio LLMs struggle on ART, with human baselines around $92.90\%$ and model performances typically well below that level. The work highlights the need for robust, cross-task audio reasoning benchmarks and outlines concrete directions for improving reasoning capabilities in multimodal models, with ART-H providing a practical human-reference subset for quick verification.

Abstract

The present benchmarks for testing the audio modality of multimodal large language models concentrate on testing various audio tasks such as speaker diarization or gender identification in isolation. Whether a multimodal model can answer the questions that require reasoning skills to combine audio tasks of different categories, cannot be verified with their use. To address this issue, we propose Audio Reasoning Tasks (ART), a new benchmark for assessing the ability of multimodal models to solve problems that require reasoning over audio signal.

A Benchmark for Audio Reasoning Capabilities of Multimodal Large Language Models

TL;DR

The paper tackles the problem of evaluating multimodal LLMs on reasoning over complex audio signals by introducing Audio Reasoning Tasks (ART), a template-driven benchmark comprising tasks that require cross-modal audio understanding and inference. ART combines question, utterance, and sound prompts generated with voice cloning and synthetic speech, enabling end-to-end assessment without leaning on single-task modules. Through two evaluation paradigms (Yes/No and Descriptive) and multiple baselines, the study demonstrates that even strong Audio LLMs struggle on ART, with human baselines around and model performances typically well below that level. The work highlights the need for robust, cross-task audio reasoning benchmarks and outlines concrete directions for improving reasoning capabilities in multimodal models, with ART-H providing a practical human-reference subset for quick verification.

Abstract

The present benchmarks for testing the audio modality of multimodal large language models concentrate on testing various audio tasks such as speaker diarization or gender identification in isolation. Whether a multimodal model can answer the questions that require reasoning skills to combine audio tasks of different categories, cannot be verified with their use. To address this issue, we propose Audio Reasoning Tasks (ART), a new benchmark for assessing the ability of multimodal models to solve problems that require reasoning over audio signal.
Paper Structure (42 sections, 96 equations, 2 figures, 16 tables)

This paper contains 42 sections, 96 equations, 2 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tasks
  4. Task Selection
  5. Task Templates
  6. Task Instances
  7. Experiments
  8. Setup
  9. Yes/No approach
  10. Descriptive approach
  11. Error analysis
  12. Benchmark Validation
  13. Conclusion
  14. Limitations
  15. ART vs. MMAU tasks
  16. ...and 27 more sections

Figures (2)

  • Figure 1: Task preparation process.
  • Figure 2: Example of a template. Both the sentence and the sound are chosen randomly from predefined lists; the target answer can be inferred from these values. Proper speakers for voice cloning for each part of the template are selected.