EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge

TL;DR

EmergentTTS-Eval introduces a six-category benchmark for evaluating TTS on emotions, paralinguistics, foreign words, questions, syntactic complexity, and complex pronunciation. It automates test-case generation with large language models and uses large audio language models as judges to provide multi-dimensional, scalable assessments. Across open- and closed-source models, the benchmark reveals nuanced strengths and systematic failures, with model judgments showing high correlation with human preferences. By releasing open-source code and data, EmergentTTS-Eval enables reproducible and extensible evaluation of expressive and robust speech synthesis.

Abstract

Text-to-Speech (TTS) benchmarks often fail to capture how well models handle nuanced and semantically complex text. Building on $\textit{EmergentTTS}$, we introduce $\textit{EmergentTTS-Eval}$, a comprehensive benchmark covering six challenging TTS scenarios: emotions, paralinguistics, foreign words, syntactic complexity, complex pronunciation (e.g. URLs, formulas), and questions. Crucially, our framework automates both test-case generation and evaluation, making the benchmark easily extensible. Starting from a small set of human-written seed prompts, we iteratively extend them using LLMs to target specific structural, phonetic and prosodic challenges, resulting in 1,645 diverse test cases. Moreover, we employ a model-as-a-judge approach, using a Large Audio Language Model (LALM) to assess the speech across multiple dimensions such as expressed emotion, prosodic, intonational, and pronunciation accuracy. We evaluate state-of-the-art open-source and proprietary TTS systems, such as 11Labs, Deepgram, and OpenAI's 4o-mini-TTS, on EmergentTTS-Eval, demonstrating its ability to reveal fine-grained performance differences. Results show that the model-as-a-judge approach offers robust TTS assessment and a high correlation with human preferences. We open source the evaluation $\href{https://github.com/boson-ai/EmergentTTS-Eval-public}{code}$ and the $\href{https://huggingface.co/datasets/bosonai/EmergentTTS-Eval}{dataset}$.

Figures (10)

• Figure 1: Paralinguistic example, refined and made more complex for TTS with increased number of cues.
• Figure 2: Foreign words example, refined and made more complex by adding idiomatically and prosodically rich foreign words.
• Figure 3: Win-rate chart for each category at different refinement depths. We also show the mean win-rate at each depth, computed collectively for high-performing models (average win-rate>50%) and low-performing models (average win-rate<50%).
• Figure 4: Left: Variance of win-rate by voice. Right: Human and model win-rate alignment.
• Figure 5: Example depth-refinement for questions category. Starting with a simple Wh-question, complexity is introduced by first by adding a subsequent Wh-question with a pragmatic nuance, then a Yes/No question to test pitch contour shifts. The final refinement examines the differentiation between interrogative and declarative prosody by inserting an emphatic statement, and further tests nuanced intonation with a concluding alternative question