Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluator

TL;DR

Vox-Evaluator tackles stability and fidelity in zero-shot TTS by introducing a multi-level evaluator that localizes erroneous speech segments, assesses overall quality, and guides both speech correction and fine-grained preference alignment. The approach combines a transformer-based architecture with DTW-driven error localization and an editing-based speech correction workflow, plus a segment-focused reward mechanism for preference optimization. To train the evaluator, the FGES dataset of synthesized speech with fine-grained pronunciation and quality annotations was created. Across multiple zero-shot TTS baselines and datasets, Vox-Evaluator improves stability and fidelity, enables efficient correction without retraining, and provides effective, fine-grained signals for alignment that reduce mispronunciations and quality artifacts.

Abstract

Recent advances in zero-shot text-to-speech (TTS), driven by language models, diffusion models and masked generation, have achieved impressive naturalness in speech synthesis. Nevertheless, stability and fidelity remain key challenges, manifesting as mispronunciations, audible noise, and quality degradation. To address these issues, we introduce Vox-Evaluator, a multi-level evaluator designed to guide the correction of erroneous speech segments and preference alignment for TTS systems. It is capable of identifying the temporal boundaries of erroneous segments and providing a holistic quality assessment of the generated speech. Specifically, to refine erroneous segments and enhance the robustness of the zero-shot TTS model, we propose to automatically identify acoustic errors with the evaluator, mask the erroneous segments, and finally regenerate speech conditioning on the correct portions. In addition, the fine-gained information obtained from Vox-Evaluator can guide the preference alignment for TTS model, thereby reducing the bad cases in speech synthesis. Due to the lack of suitable training datasets for the Vox-Evaluator, we also constructed a synthesized text-speech dataset annotated with fine-grained pronunciation errors or audio quality issues. The experimental results demonstrate the effectiveness of the proposed Vox-Evaluator in enhancing the stability and fidelity of TTS systems through the speech correction mechanism and preference optimization. The demos are shown.

Figures (5)

• Figure 1: Typical instances of distinct erroneous samples. The error segments in the generated speech are emphasized by green bounding boxes.
• Figure 2: a) Architectural overview diagram of the proposed Vox-Evaluator. b) The illustration of erroneous speech correction based on editing, the dashed box indicates that this module is optional.
• Figure 3: Performance of F5-TTS model with fine-grained preference alignment on the Seed-TTS test set at different training steps.
• Figure 4: Examples of corrected speech through speech correction.
• Figure 5: Iterative speech correction and resulting TTSDS2 evaluation failures.