Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Models

Yiming Gao, Bin Wang, Chengwei Wei, Shuo Sun, AiTi Aw

TL;DR

IFEval-Audio introduces a dedicated benchmark to study instruction following in audio-based LLMs by pairing audio inputs with structured text instructions across six dimensions and evaluating both formatting adherence and semantic correctness. The dataset contains 280 audio-instruction-answer triples drawn from speech, music, and environmental sounds, with a dual scoring framework that combines rule-based format checks and LLM-based semantic judgments. Experiments across six state-of-the-art audio LLMs reveal varying capabilities, with GPT-4o-audio and Gemini-1.5/2.0 showing stronger performance in format and content alignment, while capitalization and complex formats remain challenging. Public release of IFEval-Audio aims to drive progress in audio-text alignment and robust instruction following for multimodal AI systems.

Abstract

Large language models (LLMs) have demonstrated strong instruction-following capabilities in text-based tasks. However, this ability often deteriorates in multimodal models after alignment with non-text modalities such as images or audio. While several recent efforts have investigated instruction-following performance in text and vision-language models, instruction-following in audio-based large language models remains largely unexplored. To bridge this gap, we introduce IFEval-Audio, a novel evaluation dataset designed to assess the ability to follow instructions in an audio LLM. IFEval-Audio contains 280 audio-instruction-answer triples across six diverse dimensions: Content, Capitalization, Symbol, List Structure, Length, and Format. Each example pairs an audio input with a text instruction, requiring the model to generate an output that follows a specified structure. We benchmark state-of-the-art audio LLMs on their ability to follow audio-involved instructions. The dataset is released publicly to support future research in this emerging area.

IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Models

TL;DR

IFEval-Audio introduces a dedicated benchmark to study instruction following in audio-based LLMs by pairing audio inputs with structured text instructions across six dimensions and evaluating both formatting adherence and semantic correctness. The dataset contains 280 audio-instruction-answer triples drawn from speech, music, and environmental sounds, with a dual scoring framework that combines rule-based format checks and LLM-based semantic judgments. Experiments across six state-of-the-art audio LLMs reveal varying capabilities, with GPT-4o-audio and Gemini-1.5/2.0 showing stronger performance in format and content alignment, while capitalization and complex formats remain challenging. Public release of IFEval-Audio aims to drive progress in audio-text alignment and robust instruction following for multimodal AI systems.

Abstract

Large language models (LLMs) have demonstrated strong instruction-following capabilities in text-based tasks. However, this ability often deteriorates in multimodal models after alignment with non-text modalities such as images or audio. While several recent efforts have investigated instruction-following performance in text and vision-language models, instruction-following in audio-based large language models remains largely unexplored. To bridge this gap, we introduce IFEval-Audio, a novel evaluation dataset designed to assess the ability to follow instructions in an audio LLM. IFEval-Audio contains 280 audio-instruction-answer triples across six diverse dimensions: Content, Capitalization, Symbol, List Structure, Length, and Format. Each example pairs an audio input with a text instruction, requiring the model to generate an output that follows a specified structure. We benchmark state-of-the-art audio LLMs on their ability to follow audio-involved instructions. The dataset is released publicly to support future research in this emerging area.

Paper Structure

This paper contains 19 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. IFEval-Audio
  4. Evaluation Dimensions
  5. Audio Sources and Diversity
  6. Dataset Structure
  7. Annotation Process
  8. Evaluation Design
  9. Experiments
  10. Models Tested
  11. Results, Analysis and Discussions
  12. Conclusion
  13. Limitations
  14. Additional Related Work
  15. Appendix Figures
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Breakdown of instruction subcategories within the dimensions, with approximately 25% Content, 20% Capitalization, 15% Symbol, 15% List Structure, 10% Length, and 15% Format requirements.
  • Figure 2: Distribution of triples across dimensions for Speech, Music, and Environmental Sound audio.