S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information

TL;DR

S2S-Arena addresses the gap in evaluating instruction-following for Speech2Speech models when paralinguistic information is present by introducing an arena-style benchmark with 154 samples across four domains and 21 tasks. It employs an ELO-based, reference-free, manual comparison framework to assess four model classes, revealing that GPT-4o-based systems deliver the strongest performance, while cascaded ASR-LLM-TTS pipelines also excel in knowledge-rich tasks. The study finds that the LLM backbone largely drives multilingual capabilities, while the speech module constraints (e.g., codecs) limit language coverage; paralinguistic generation remains a key challenge. It also uncovers biases in speech-based evaluation, such as positional and length effects, and demonstrates that using speech models as automatic judges yields low agreement with human judgments, underscoring the need for robust automatic evaluation frameworks for paralinguistic-aware S2S systems.

Abstract

The rapid development of large language models (LLMs) has brought significant attention to speech models, particularly recent progress in speech2speech protocols supporting speech input and output. However, the existing benchmarks adopt automatic text-based evaluators for evaluating the instruction following ability of these models lack consideration for paralinguistic information in both speech understanding and generation. To address these issues, we introduce S2S-Arena, a novel arena-style S2S benchmark that evaluates instruction-following capabilities with paralinguistic information in both speech-in and speech-out across real-world tasks. We design 154 samples that fused TTS and live recordings in four domains with 21 tasks and manually evaluate existing popular speech models in an arena-style manner. The experimental results show that: (1) in addition to the superior performance of GPT-4o, the speech model of cascaded ASR, LLM, and TTS outperforms the jointly trained model after text-speech alignment in speech2speech protocols; (2) considering paralinguistic information, the knowledgeability of the speech model mainly depends on the LLM backbone, and the multilingual support of that is limited by the speech module; (3) excellent speech models can already understand the paralinguistic information in speech input, but generating appropriate audio with paralinguistic information is still a challenge.

Figures (4)

• Figure 1: An Example of Evaluating Instruction Following with Rhythm Controlling in Speech-in and Speech-out for Speech Models.
• Figure 2: The Three-Stage Process of S2S-Arena Construction: Task Determination, Instruction Design and Instruction Recording.
• Figure 3: The Evaluation Process of S2S-Arena.
• Figure 4: Pair-wise comparison of various models.