Position: Towards Responsible Evaluation for Text-to-Speech

TL;DR

This paper addresses the mismatch between rapid Text-to-Speech (TTS) advancements and current evaluation practices, highlighting risks from high-fidelity synthesis such as deepfakes and privacy violations. It proposes Responsible Evaluation, a three-level framework: Level One strengthens fidelity and accuracy with robust objective/subjective scoring and expanded evaluation dimensions; Level Two enforces comparability, standardization, and transferability through standardized datasets, transparent reporting, and transferable metrics; Level Three introduces ethical and risk oversight addressing data provenance, traceability, bias, and misuse risk. The work provides a critical diagnosis of existing practices, actionable recommendations for each level, and discusses alternative viewpoints on complexity and innovation speed. Its significance lies in guiding the development of trustworthy, ethically-aligned TTS systems that balance technical progress with societal interests and safety considerations.

Abstract

Recent advances in text-to-speech (TTS) technology have enabled systems to generate speech that is often indistinguishable from human speech, bringing benefits to accessibility, content creation, and human-computer interaction. However, current evaluation practices are increasingly inadequate for capturing the full range of capabilities, limitations, and societal impacts of modern TTS systems. This position paper introduces the concept of Responsible Evaluation and argues that it is essential and urgent for the next phase of TTS development, structured through three progressive levels: (1) ensuring the faithful and accurate reflection of a model's true capabilities and limitations, with more robust, discriminative, and comprehensive objective and subjective scoring methodologies; (2) enabling comparability, standardization, and transferability through standardized benchmarks, transparent reporting, and transferable evaluation metrics; and (3) assessing and mitigating ethical risks associated with forgery, misuse, privacy violations, and security vulnerabilities. Through this concept, we critically examine current evaluation practices, identify systemic shortcomings, and propose actionable recommendations. We hope this concept will not only foster more reliable and trustworthy TTS technology but also guide its development toward ethically sound and societally beneficial applications.

Figures (1)

• Figure 1: Co-evolution of TTS technology and TTS evaluation across three phases.