Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese

TL;DR

The paper tackles the challenge of evaluating human-likeness in large-language-model–based TTS for Chinese by proposing Audio Turing Test (ATT), a multidimensional corpus (ATT-Corpus) paired with a Turing-test–inspired protocol to assess human-likeness rather than rely on MOS. It also trains Auto-ATT, a LoRA-tuned Qwen2-Audio-Instruct model, to predict the Human-likeness Score (HLS) for rapid automatic evaluation. Experiments with 20 voices across 5 model families and 857 native Chinese listeners show ATT can differentiate models across linguistic and paralinguistic dimensions, with Seed-TTS achieving the top HLS yet still far from human performance. Auto-ATT aligns strongly with human judgments and outperforms MOS-based predictors on trap items, enabling fast, reliable evaluation to accelerate TTS development. The ATT framework and Auto-ATT data/tools are publicly available (ATT Hugging Face Collection) to promote reproducibility and broader adoption.

Abstract

Recent advances in large language models (LLMs) have significantly improved text-to-speech (TTS) systems, enhancing control over speech style, naturalness, and emotional expression, which brings TTS Systems closer to human-level performance. Although the Mean Opinion Score (MOS) remains the standard for TTS System evaluation, it suffers from subjectivity, environmental inconsistencies, and limited interpretability. Existing evaluation datasets also lack a multi-dimensional design, often neglecting factors such as speaking styles, context diversity, and trap utterances, which is particularly evident in Chinese TTS evaluation. To address these challenges, we introduce the Audio Turing Test (ATT), a multi-dimensional Chinese corpus dataset ATT-Corpus paired with a simple, Turing-Test-inspired evaluation protocol. Instead of relying on complex MOS scales or direct model comparisons, ATT asks evaluators to judge whether a voice sounds human. This simplification reduces rating bias and improves evaluation robustness. To further support rapid model development, we also finetune Qwen2-Audio-Instruct with human judgment data as Auto-ATT for automatic evaluation. Experimental results show that ATT effectively differentiates models across specific capability dimensions using its multi-dimensional design. Auto-ATT also demonstrates strong alignment with human evaluations, confirming its value as a fast and reliable assessment tool. The white-box ATT-Corpus and Auto-ATT can be found in ATT Hugging Face Collection (https://huggingface.co/collections/meituan/audio-turing-test-682446320368164faeaf38a4).

Figures (4)

• Figure 1: Audio Turing Test Evaluation Framework: (1) Corpus Generation: a semi-automatic corpus generation pipeline for generating the challenge TTS synthesis corpus for ATT evaluation; (2) Human Evaluation: a human-evaluation protocol that enables precise, comparable assessments and lowers evaluation costs through a simple yet effective Turing-test-style design, (3) Automatic Evaluation: Auto-ATT, an automatic tool to predict the Human-likeness Score for rapid iterations.
• Figure 2: The Key Benchmark Results of ATT Human Evaluation.
• Figure 3: The prediction results of Trap Item through DMSMOS Pro, UTMOSv2, and Auto-ATT. For a human speech clip, the ideal outcome is a true positive: the red dot should fall within the red zone; for a flawed synthetic speech clip, the ideal outcome is a true negative: the gray dot should fall within the gray zone.
• Figure 4: The Screen of One Audio Clip in ATT Evaluation.