Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot vs. Few-Shot Multi-Speaker TTS Using Pre-trained Czech SpeechT5 Model

Jan Lehečka, Zdeněk Hanzlíček, Jindřich Matoušek, Daniel Tihelka

TL;DR

The paper investigates zero-shot versus few-shot multi-speaker TTS using a Czech SpeechT5 foundation model trained from scratch on large-scale Czech speech and text data. It pre-trains on approximately 120,000 hours of Czech speech and 17.5 billion Czech text tokens, then fine-tunes on a large, diverse multi-speaker corpus, leveraging x-vector speaker embeddings and a HiFi-GAN vocoder for synthesis. Two listening tests show zero-shot synthesis underperforms in both quality and speaker similarity, while roughly 1 minute of target speaker data yields acceptable quality, with diminishing quality gains as data increases but potential gains in similarity for less monotonous voices. The work demonstrates practical viability for Czech zero-/few-shot TTS, discusses ethical considerations for synthetic voices of public figures, and provides the pre-trained Czech SpeechT5 foundation model for non-commercial use.

Abstract

In this paper, we experimented with the SpeechT5 model pre-trained on large-scale datasets. We pre-trained the foundation model from scratch and fine-tuned it on a large-scale robust multi-speaker text-to-speech (TTS) task. We tested the model capabilities in a zero- and few-shot scenario. Based on two listening tests, we evaluated the synthetic audio quality and the similarity of how synthetic voices resemble real voices. Our results showed that the SpeechT5 model can generate a synthetic voice for any speaker using only one minute of the target speaker's data. We successfully demonstrated the high quality and similarity of our synthetic voices on publicly known Czech politicians and celebrities.

Zero-Shot vs. Few-Shot Multi-Speaker TTS Using Pre-trained Czech SpeechT5 Model

TL;DR

The paper investigates zero-shot versus few-shot multi-speaker TTS using a Czech SpeechT5 foundation model trained from scratch on large-scale Czech speech and text data. It pre-trains on approximately 120,000 hours of Czech speech and 17.5 billion Czech text tokens, then fine-tunes on a large, diverse multi-speaker corpus, leveraging x-vector speaker embeddings and a HiFi-GAN vocoder for synthesis. Two listening tests show zero-shot synthesis underperforms in both quality and speaker similarity, while roughly 1 minute of target speaker data yields acceptable quality, with diminishing quality gains as data increases but potential gains in similarity for less monotonous voices. The work demonstrates practical viability for Czech zero-/few-shot TTS, discusses ethical considerations for synthetic voices of public figures, and provides the pre-trained Czech SpeechT5 foundation model for non-commercial use.

Abstract

In this paper, we experimented with the SpeechT5 model pre-trained on large-scale datasets. We pre-trained the foundation model from scratch and fine-tuned it on a large-scale robust multi-speaker text-to-speech (TTS) task. We tested the model capabilities in a zero- and few-shot scenario. Based on two listening tests, we evaluated the synthetic audio quality and the similarity of how synthetic voices resemble real voices. Our results showed that the SpeechT5 model can generate a synthetic voice for any speaker using only one minute of the target speaker's data. We successfully demonstrated the high quality and similarity of our synthetic voices on publicly known Czech politicians and celebrities.
Paper Structure (15 sections, 3 figures, 1 table)

This paper contains 15 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. SpeechT5
  3. Pre-training
  4. Speech Data
  5. Text Data
  6. Pre-training Setup
  7. Fine-tuning
  8. Data Preprocessing
  9. Fine-tuning Setup
  10. Few-Shot Speech Data Collection
  11. Results and Discussion
  12. Listening tests
  13. Conclusion
  14. Acknowledgments.
  15. Disclosure of Interests.

Figures (3)

  • Figure 1: Results of the quality listening test. We show the results for the original real-human records (orig) and for the records generated using the SpeechT5 model: the zero-shot approach (zs) and few-shot approaches using 10 seconds of training data (10s), one minute (1m), and five minutes (5m) respectively.
  • Figure 2: Quality listening test -- results for individual voices.
  • Figure 3: Results of the similarity listening test.