Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversion

TL;DR

This work tackles zero-shot voice conversion by disentangling content and timbre using semantic tokens obtained from self-supervised models and enabling reference-based timbre transfer through in-context learning (ICL). It introduces ICL-VC, a mask-and-reconstruction framework built on flow-matching generation, with semantic tokenization via a universal 500-cluster k-means across HuBERT, Wav2Vec, and XLSR, and optional prosody embeddings from Emotion2Vec to improve prosody preservation. Key findings show that ICL-VC substantially enhances speaker similarity and naturalness compared to baselines, demonstrates the generality of k-means tokenization across models, and indicates that emotion-driven prosody embeddings effectively mitigate prosody leakage while maintaining timbre quality. The approach negates the need for a pre-trained speaker encoder and shows promise for scaling to larger, noisier, and multilingual data, with future work focusing on further improving prosody preservation and evaluating on broader datasets.

Abstract

Voice conversion (VC) aims to modify the speaker's timbre while retaining speech content. Previous approaches have tokenized the outputs from self-supervised into semantic tokens, facilitating disentanglement of speech content information. Recently, in-context learning (ICL) has emerged in text-to-speech (TTS) systems for effectively modeling specific characteristics such as timbre through context conditioning. This paper proposes an ICL capability enhanced VC system (ICL-VC) employing a mask and reconstruction training strategy based on flow-matching generative models. Augmented with semantic tokens, our experiments on the LibriTTS dataset demonstrate that ICL-VC improves speaker similarity. Additionally, we find that k-means is a versatile tokenization method applicable to various pre-trained models. However, the ICL-VC system faces challenges in preserving the prosody of the source speech. To mitigate this issue, we propose incorporating prosody embeddings extracted from a pre-trained emotion recognition model into our system. Integration of prosody embeddings notably enhances the system's capability to preserve source speech prosody, as validated on the Emotional Speech Database.

Figures (2)

• Figure 1: System Overview.
• Figure 2: The relationship between the speaker embedding cosine similarity (SECS) and reference speech duration.