Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speech

Karl El Hajal, Ajinkya Kulkarni, Enno Hermann, Mathew Magimai. -Doss

TL;DR

The paper tackles the data efficiency challenge of zero-shot multi-speaker TTS by introducing kNN-TTS, a retrieval-based framework that uses SSL feature representations to transfer voice style from a target speaker without requiring multi-speaker transcribed data. Text-to-SSL models generate source SSL features from text, which are then replaced frame-by-frame via kNN retrieval from a target speaker's SSL unit database, followed by a linear interpolation controlled by the parameter $λ$ to morph voice characteristics. A pre-trained vocoder reconstructs the waveform from the converted SSL features, enabling high-quality speech with strong speaker similarity. The approach demonstrates competitive performance with state-of-the-art baselines trained on much larger datasets, while offering significant gains in data efficiency and enabling fine-grained voice morphing, making it suitable for low-resource languages and domains.

Abstract

While recent zero-shot multi-speaker text-to-speech (TTS) models achieve impressive results, they typically rely on extensive transcribed speech datasets from numerous speakers and intricate training pipelines. Meanwhile, self-supervised learning (SSL) speech features have emerged as effective intermediate representations for TTS. Further, SSL features from different speakers that are linearly close share phonetic information while maintaining individual speaker identity. In this study, we introduce kNN-TTS, a simple and effective framework for zero-shot multi-speaker TTS using retrieval methods which leverage the linear relationships between SSL features. Objective and subjective evaluations show that our models, trained on transcribed speech from a single speaker only, achieve performance comparable to state-of-the-art models that are trained on significantly larger training datasets. The low training data requirements mean that kNN-TTS is well suited for the development of multi-speaker TTS systems for low-resource domains and languages. We also introduce an interpolation parameter which enables fine-grained voice morphing. Demo samples are available at https://idiap.github.io/knn-tts

kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speech

TL;DR

The paper tackles the data efficiency challenge of zero-shot multi-speaker TTS by introducing kNN-TTS, a retrieval-based framework that uses SSL feature representations to transfer voice style from a target speaker without requiring multi-speaker transcribed data. Text-to-SSL models generate source SSL features from text, which are then replaced frame-by-frame via kNN retrieval from a target speaker's SSL unit database, followed by a linear interpolation controlled by the parameter to morph voice characteristics. A pre-trained vocoder reconstructs the waveform from the converted SSL features, enabling high-quality speech with strong speaker similarity. The approach demonstrates competitive performance with state-of-the-art baselines trained on much larger datasets, while offering significant gains in data efficiency and enabling fine-grained voice morphing, making it suitable for low-resource languages and domains.

Abstract

While recent zero-shot multi-speaker text-to-speech (TTS) models achieve impressive results, they typically rely on extensive transcribed speech datasets from numerous speakers and intricate training pipelines. Meanwhile, self-supervised learning (SSL) speech features have emerged as effective intermediate representations for TTS. Further, SSL features from different speakers that are linearly close share phonetic information while maintaining individual speaker identity. In this study, we introduce kNN-TTS, a simple and effective framework for zero-shot multi-speaker TTS using retrieval methods which leverage the linear relationships between SSL features. Objective and subjective evaluations show that our models, trained on transcribed speech from a single speaker only, achieve performance comparable to state-of-the-art models that are trained on significantly larger training datasets. The low training data requirements mean that kNN-TTS is well suited for the development of multi-speaker TTS systems for low-resource domains and languages. We also introduce an interpolation parameter which enables fine-grained voice morphing. Demo samples are available at https://idiap.github.io/knn-tts
Paper Structure (15 sections, 1 equation, 3 figures, 3 tables)

This paper contains 15 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Approach
  3. Framework
  4. Implementation
  5. Experimental Setup
  6. Baselines
  7. Evaluation
  8. Results and analysis
  9. Discussion and conclusions
  10. Spectral Features
  11. Model and Training Details
  12. Baselines Details
  13. Listening Test
  14. Naturalness:
  15. Similarity:

Figures (3)

  • Figure 1: kNN-TTS framework overview. Only the Text-to-SSL model is trained on transcribed audio. The SSL encoder, vocoder are pre-trained on untranscribed multi-speaker data, and the kNN algorithm is non-parametric.
  • Figure 2: Speaker similarity matrix comparing SECS values for ground truth (GT) LJSpeech samples, LibriSpeech Speaker 4077 (Libri4077) recordings, and GlowkNN-TTS outputs with kNN retrieval from Libri4077 data for various $\lambda$ values. Samples in each case are split in half into sets $A$ and $B$ and compared.
  • Figure 3: (a) Mean UTMOS ($\uparrow$) and WER ($\downarrow$) for kNN-TTS outputs using different amounts of LJSpeech reference utterances. (b) Mean SECS ($\uparrow$) and WER ($\downarrow$) for kNN-TTS and baseline outputs using different amounts of LibriSpeech Speaker 4077 reference utterances.