MulliVC: Multi-lingual Voice Conversion With Cycle Consistency

TL;DR

MulliVC tackles the challenge of multi-lingual voice conversion without requiring paired multi-lingual data by introducing a three-step cycle-training scheme. The method explicitly disentangles timbre from content and prosody, using cross-language steps 2 and 3 to enforce timbre-only transfer, aided by a Fine-grained Timbre Conformer and auxiliary losses including ASR perceptual loss and a timbre loss from a speaker verification model. Empirical results across English, Mandarin, and bilingual settings show state-of-the-art or near state-of-the-art performance in both monolingual and cross-lingual zero-shot VC, with strong improvements in speaker similarity and intelligibility. The work demonstrates the viability of cycle-consistent, language-agnostic timbre transfer and provides a path toward robust cross-language VC without extensive multilingual paired data.

Abstract

Voice conversion aims to modify the source speaker's voice to resemble the target speaker while preserving the original speech content. Despite notable advancements in voice conversion these days, multi-lingual voice conversion (including both monolingual and cross-lingual scenarios) has yet to be extensively studied. It faces two main challenges: 1) the considerable variability in prosody and articulation habits across languages; and 2) the rarity of paired multi-lingual datasets from the same speaker. In this paper, we propose MulliVC, a novel voice conversion system that only converts timbre and keeps original content and source language prosody without multi-lingual paired data. Specifically, each training step of MulliVC contains three substeps: In step one the model is trained with monolingual speech data; then, steps two and three take inspiration from back translation, construct a cyclical process to disentangle the timbre and other information (content, prosody, and other language-related information) in the absence of multi-lingual data from the same speaker. Both objective and subjective results indicate that MulliVC significantly surpasses other methods in both monolingual and cross-lingual contexts, demonstrating the system's efficacy and the viability of the three-step approach with cycle consistency. Audio samples can be found on our demo page (mullivc.github.io).

Figures (6)

• Figure 1: Training of MilliVC. SPK_1|LAN_A|#2 denotes speech#2 said by speaker 1 who speaks language A.
• Figure 2: Model architecture of MulliVC. Note that modules printed with a lock are frozen when training. We use Ⓒ,ⒸT to denote concatenate along the channel axis and concatenate along the time axis respectively
• Figure 3: The Fine-grained Timbre Conformer architecture.
• Figure 4: Si_F, Sj_M denotes female speaker i and male speaker j respectively. Speeches of Chinese Mandarin spoken -by the corresponding speaker are displayed on the horizontal axis. Speeches of English are displayed on the vertical axis. The number in each grid is the average SIM between the speeches.
• Figure 5: Visualization of the speaker embedding space based on t-SNE and four randomly selected speakers in the EMIME dataset. spk$i$_MAN denotes the Chinese Mandarin speech of speaker $i$. And spk$i$_ENG denotes the English speech of speaker $i$.