StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching

TL;DR

StableVC addresses the gap in zero-shot voice conversion by enabling independent control of timbre and style for unseen speakers using a non-autoregressive conditional flow matching framework. It disentangles content, timbre, and style with three extractors and leverages a DualAGC mechanism within DiT blocks to capture nuanced timbre and style, guided by a timbre prior and an adaptive style gate. Training combines flow-matching objectives with duration alignment and adversarial GRL to reduce timbre leakage, achieving fast generation with 10 Euler steps and competitive quality. Empirically, StableVC outperforms state-of-the-art baselines on zero-shot VC and style transfer tasks and delivers substantial speedups (approximately 25× vs autoregressive and 1.65× vs diffusion-based methods), demonstrating strong practical applicability for high-quality, controllable speech synthesis.

Abstract

Zero-shot voice conversion (VC) aims to transfer the timbre from the source speaker to an arbitrary unseen speaker while preserving the original linguistic content. Despite recent advancements in zero-shot VC using language model-based or diffusion-based approaches, several challenges remain: 1) current approaches primarily focus on adapting timbre from unseen speakers and are unable to transfer style and timbre to different unseen speakers independently; 2) these approaches often suffer from slower inference speeds due to the autoregressive modeling methods or the need for numerous sampling steps; 3) the quality and similarity of the converted samples are still not fully satisfactory. To address these challenges, we propose a style controllable zero-shot VC approach named StableVC, which aims to transfer timbre and style from source speech to different unseen target speakers. Specifically, we decompose speech into linguistic content, timbre, and style, and then employ a conditional flow matching module to reconstruct the high-quality mel-spectrogram based on these decomposed features. To effectively capture timbre and style in a zero-shot manner, we introduce a novel dual attention mechanism with an adaptive gate, rather than using conventional feature concatenation. With this non-autoregressive design, StableVC can efficiently capture the intricate timbre and style from different unseen speakers and generate high-quality speech significantly faster than real-time. Experiments demonstrate that our proposed StableVC outperforms state-of-the-art baseline systems in zero-shot VC and achieves flexible control over timbre and style from different unseen speakers. Moreover, StableVC offers approximately 25x and 1.65x faster sampling compared to autoregressive and diffusion-based baselines.

Figures (4)

• Figure 1: The concept of style-controllable zero-shot voice conversion. It aims to build a VC system capable of adapting timbre to unseen speakers and transferring the style to another unseen speaker. Here, "unseen" refers to speakers not present in the training set.
• Figure 2: The overall framework of StableVC includes three feature extractors for style, linguistic content, and mel-spectrogram extraction. It also incorporates a content module and a duration module to re-predict the duration based on different styles and timbres, and a flow matching module generates high-quality speech at speeds significantly faster than real-time.
• Figure 3: Details of DualAGC in the DiT block.
• Figure 4: Violin plots for timbre and style similarity of speech generated by baseline systems and StableVC.