Zero-shot Cross-lingual Voice Transfer for TTS
Fadi Biadsy, Youzheng Chen, Isaac Elias, Kyle Kastner, Gary Wang, Andrew Rosenberg, Bhuvana Ramabhadran
TL;DR
This work presents a zero-shot cross-lingual voice transfer module that can be integrated into a multi-lingual TTS system using a single reference utterance. The VT module comprises a speaker encoder, a bottleneck layer, and residual adapters, with four bottleneck variants (VAE, SharedGST, MultiGST, SegmentGST) explored for their impact on MOS and speaker fidelity across nine languages. Across typical speech, SegmentGST achieves the best overall voice transfer quality with an average MOS around 3.9 and 73% speaker-similarity, while for atypical speech (dysarthria), SharedGST and MultiGST deliver higher intelligibility and speaker fidelity, enabling voice restoration in cases with non-typical input. A watermarking approach is proposed to mitigate potential misuse, and case studies demonstrate practical voice restoration utilities for individuals who have never had typical speech. The work highlights the modularity and cross-lingual capabilities of zero-shot VT and its potential impact on accessibility and personalized TTS across languages.
Abstract
In this paper, we introduce a zero-shot Voice Transfer (VT) module that can be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to transfer an individual's voice across languages. Our proposed VT module comprises a speaker-encoder that processes reference speech, a bottleneck layer, and residual adapters, connected to preexisting TTS layers. We compare the performance of various configurations of these components and report Mean Opinion Score (MOS) and Speaker Similarity across languages. Using a single English reference speech per speaker, we achieve an average voice transfer similarity score of 73% across nine target languages. Vocal characteristics contribute significantly to the construction and perception of individual identity. The loss of one's voice, due to physical or neurological conditions, can lead to a profound sense of loss, impacting one's core identity. As a case study, we demonstrate that our approach can not only transfer typical speech but also restore the voices of individuals with dysarthria, even when only atypical speech samples are available - a valuable utility for those who have never had typical speech or banked their voice. Cross-lingual typical audio samples, plus videos demonstrating voice restoration for dysarthric speakers are available here (google.github.io/tacotron/publications/zero_shot_voice_transfer).