Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang, Ming Zhu, Juntao Tan, Haolin Chen, Wenting Zhao, Rithesh Murthy, Roshan Ram, Akshara Prabhakar, Shelby Heinecke, Caiming, Xiong, Silvio Savarese, Huan Wang

TL;DR

The results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates.

Abstract

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: \textit{accuracy} (semantic correctness), \textit{completeness} (coverage of reference content), and \textit{hallucination} (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

TL;DR

The results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates.

Abstract

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: \textit{accuracy} (semantic correctness), \textit{completeness} (coverage of reference content), and \textit{hallucination} (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.
Paper Structure (35 sections, 2 equations, 4 figures, 3 tables)

This paper contains 35 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. AudioCapBench
  3. Dataset Construction
  4. Evaluation Metrics
  5. LLM-as-Judge (Primary).
  6. Reference-Based Metrics (Secondary).
  7. Models Evaluated
  8. Results
  9. Overall Leaderboard
  10. Per-Category Analysis
  11. Findings
  12. Accuracy--Hallucination Trade-off.
  13. Realtime Models Underperform.
  14. Gemini's Completeness Advantage.
  15. Reference Metrics vs. LLM Judge.
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Distribution of audio duration (top) and reference caption length (bottom) for each category. Dashed lines indicate means. Music clips are longest; speech clips are shortest but have detailed emotional captions.
  • Figure 2: Overall LLM judge scores for all 13 models, colored by provider. Gemini models (blue) dominate the top positions, followed by OpenAI Chat Completions (green) and Realtime (orange).
  • Figure 3: Per-category LLM Overall scores. Speech is consistently the easiest category across all models, while music is the hardest. Performance gaps between models are largest on sound captioning.
  • Figure 4: Accuracy vs. hallucination trade-off. Models in the upper-right are both accurate and grounded. OpenAI mini models cluster in the upper-left (conservative but empty). Gemini models tend toward higher accuracy but lower hallucination scores.