Voice Cloning: Comprehensive Survey

TL;DR

This paper surveys voice cloning, defined as extending TTS to imitate a target speaker's voice while balancing data efficiency and naturalness. It establishes standardized terminology and taxonomy, organizing methods into speaker adaptation, few-shot, zero-shot, and multilingual TTS. The survey covers foundational TTS components, pre-training, adversarial and meta-learning approaches, and a wide range of voice-cloning architectures including encoder–decoder, diffusion, and codec-based models. It also documents datasets and evaluation metrics and discusses applications and potential misuse, offering directions for benchmark development and detection to mitigate audio deepfakes.

Abstract

Voice Cloning has rapidly advanced in today's digital world, with many researchers and corporations working to improve these algorithms for various applications. This article aims to establish a standardized terminology for voice cloning and explore its different variations. It will cover speaker adaptation as the fundamental concept and then delve deeper into topics such as few-shot, zero-shot, and multilingual TTS within that context. Finally, we will explore the evaluation metrics commonly used in voice cloning research and related datasets. This survey compiles the available voice cloning algorithms to encourage research toward its generation and detection to limit its misuse.

Figures (7)

• Figure 1: Bar graph showing the trend in domains related to speaker adaptation.
• Figure 2: Common TTS algorithms, adapted from Tacotron 2 Shen_Tacotron2_ICASSP2018, FastSpeech 2 Ren_FastSpeech2_ICLR2021, VITS Kim_VITS_ICML2021, and VALL-E Wang_VALLE_arxiv2023.
• Figure 3: Overview of the Survey paper.
• Figure 4: Four variants for speaker adaptation of a Multi-Speaker TTS.
• Figure 5: Few-Shot TTS algorithms, adapted from AdaSpeech Adaspeech_ICLR2021, AdaSpeech 2 Adaspeech2_ICASSP2021, AdaSpeech 3 AdaSpeech3_InterSpeech2021, and Cai et al. Cai_InterSpeech2020.