URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models

TL;DR

URO-Bench introduces a comprehensive S2S benchmark for end-to-end spoken dialogue systems, addressing cognitive and paralinguistic abilities in multilingual, multi-round contexts. The framework uses a two-track design with 40 datasets, a rigorous data construction pipeline, and four evaluation metrics to holistically assess understanding, reasoning, and oral output quality. Experimental results reveal that open-source SDMs lag behind backbone LLMs and proprietary systems in instruction-following, reasoning, multilinguality, and paralinguistics, highlighting key directions for future research. The authors provide open-source data, code, and a leaderboard to accelerate progress in the field. Overall, URO-Bench represents a significant step toward standardized, multifaceted evaluation of end-to-end S2S models with practical implications for real-world voice-enabled AI systems.

Abstract

Recent advances in large language models (LLMs) have driven significant progress in end-to-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.

Figures (7)

• Figure 1: Overview of URO-Bench. Chart (a) and (b) demonstrate all the datasets for the basic track and pro track respectively. Chart (c) is the capability radar chart of 6 open-source SDMs and GPT-4o-Audio-Preview on English proficiency. For the Chinese capability radar chart, please refer to \ref{['fig:radar_zh']}.
• Figure 2: Representative examples illustrating the taxonomy of URO-Bench. Covering a diverse range of s2s tasks across the dimensions of understanding, reasoning, and oral conversation, URO-Bench is able to reflect a spoken dialogue model's abilities comprehensively.
• Figure 3: URO-Bench Benchmark Construction Pipeline. We performed manual reviews twice, one in Data Filtering for meta-data quality and another in ASR & Manual Review for speech quality. By adhering to a systematic and disciplined approach, we ensured that the datasets are diverse, comprehensive, and of high quality.
• Figure 4: Results of consistency between task accomplish scores and human evaluations.
• Figure 5: Capability radar chart of 4 SDMs on Chinese proficiency.