YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

TL;DR

YourTTS proposes a multilingual, zero-shot extension of VITS for simultaneous zero-shot multi-speaker TTS and zero-shot voice conversion. It introduces raw-text input, language-conditioned embeddings, a deep transformer encoder, a flow-based decoder with a VAE posterior encoder, and external speaker conditioning to achieve cross-speaker and cross-language synthesis with end-to-end training. The approach attains SOTA results on VCTK for ZS-TTS, demonstrates competitive zero-shot VC across languages, and enables effective speaker adaptation with less than one minute of speech, highlighting practical utility for low-resource languages. Limitations include duration-prediction instability and gender-imbalanced data, suggesting directions for robust duration modeling and broader multilingual data collection.

Abstract

YourTTS brings the power of a multilingual approach to the task of zero-shot multi-speaker TTS. Our method builds upon the VITS model and adds several novel modifications for zero-shot multi-speaker and multilingual training. We achieved state-of-the-art (SOTA) results in zero-shot multi-speaker TTS and results comparable to SOTA in zero-shot voice conversion on the VCTK dataset. Additionally, our approach achieves promising results in a target language with a single-speaker dataset, opening possibilities for zero-shot multi-speaker TTS and zero-shot voice conversion systems in low-resource languages. Finally, it is possible to fine-tune the YourTTS model with less than 1 minute of speech and achieve state-of-the-art results in voice similarity and with reasonable quality. This is important to allow synthesis for speakers with a very different voice or recording characteristics from those seen during training.

Figures (1)

• Figure 1: YourTTS diagram depicting (a) training procedure and (b) inference procedure.