HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts

TL;DR

HybridVC addresses the need for flexible, data-efficient voice style conversion by combining a pre-trained CVAE backbone with a contrastive learning stage that aligns text embeddings with speaker style. It supports both audio and text prompts by refining a text embedding $g$ in parallel with a speaker style embedding $s$, using a KL-based latent model and a contrastive loss. The approach achieves competitive intelligibility, naturalness, and audio quality with substantially reduced training time (e.g., ~15 hours on RTX3090) and demonstrates robustness across datasets like VCTK and PromptSpeech. This enables practical applications such as user-defined personalized voices on social media, while ablations confirm the value of the latent model and negative sampling for text alignment, albeit with some sensitivity differences for pure text prompts.

Abstract

We introduce HybridVC, a voice conversion (VC) framework built upon a pre-trained conditional variational autoencoder (CVAE) that combines the strengths of a latent model with contrastive learning. HybridVC supports text and audio prompts, enabling more flexible voice style conversion. HybridVC models a latent distribution conditioned on speaker embeddings acquired by a pretrained speaker encoder and optimises style text embeddings to align with the speaker style information through contrastive learning in parallel. Therefore, HybridVC can be efficiently trained under limited computational resources. Our experiments demonstrate HybridVC's superior training efficiency and its capability for advanced multi-modal voice style conversion. This underscores its potential for widespread applications such as user-defined personalised voice in various social media platforms. A comprehensive ablation study further validates the effectiveness of our method.

Figures (6)

• Figure 1: Overview of HybridVC, where $z$ represents the latent variable obtained by CVAE backbone, $g$ is text embeddings, $s$ is speaker style embeddings, and $\{g^1,...,g^N\}$ are $N$ negative samples. The snowflake symbol represents frozen parameters during training.
• Figure 2: Illustration of negative sampling for text embedding.
• Figure 3: Illustration of optimising the text embedding $g^*$.
• Figure 4: Visualization of source speech (left) and converted speech with text prompt "he speaks loudly" (right).
• Figure 5: Cosine similarity distribution before text embedding optimisation (left-top); after fine-tuned with proposed negative sampling method (right-top); after fine-tuned with negative samples within the batch (left-bottom); after fine-tuned with negative samples with manually classified labels (right-bottom).