UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models

TL;DR

UltraEval-Audio introduces the first unified framework for evaluating audio foundation models across understanding, generation, and codecs, addressing fragmented data sources, codec evaluation gaps, and English-centric benchmarks. Its modular, configuration-driven design decouples data, prompts, inference, and post-processing, enabling one-command evaluations and real-time leaderboards. The framework adds a three-dimensional codec evaluation and two Chinese benchmarks (SpeechCMMLU and SpeechHSK) to broaden linguistic coverage and diagnostic depth. Empirical results across 13 models and 9 codecs on three leaderboards demonstrate nuanced trade-offs in ASR accuracy, generation quality, and timbre fidelity, providing actionable insights for researchers and industry practitioners toward fair, reproducible, and transparent comparisons.

Abstract

The development of audio foundation models has accelerated rapidly since the emergence of GPT-4o. However, the lack of comprehensive evaluation has become a critical bottleneck for further progress in the field, particularly in audio generation. Current audio evaluation faces three major challenges: (1) audio evaluation lacks a unified framework, with datasets and code scattered across various sources, hindering fair and efficient cross-model comparison;(2) audio codecs, as a key component of audio foundation models, lack a widely accepted and holistic evaluation methodology; (3) existing speech benchmarks are heavily reliant on English, making it challenging to objectively assess models' performance on Chinese. To address the first issue, we introduce UltraEval-Audio, a unified evaluation framework for audio foundation models, specifically designed for both audio understanding and generation tasks. UltraEval-Audio features a modular architecture, supporting 10 languages and 14 core task categories, while seamlessly integrating 24 mainstream models and 36 authoritative benchmarks. To enhance research efficiency, the framework provides a one-command evaluation feature, accompanied by real-time public leaderboards. For the second challenge, UltraEval-Audio adopts a novel comprehensive evaluation scheme for audio codecs, evaluating performance across three key dimensions: semantic accuracy, timbre fidelity, and acoustic quality. To address the third issue, we propose two new Chinese benchmarks, SpeechCMMLU and SpeechHSK, designed to assess Chinese knowledge proficiency and language fluency. We wish that UltraEval-Audio will provide both academia and industry with a transparent, efficient, and fair platform for comparison of audio models. Our code, benchmarks, and leaderboards are available at https://github.com/OpenBMB/UltraEval-Audio.

Figures (2)

• Figure 1: Overview of the UltraEval-Audio framework
• Figure 2: Mimi family performance comparison. Note that ASR-WER is normalized as ($100 - \text{WER}$), and speech quality scores (acoustic scores) are scaled by a factor of 20 for visualization.