Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

TL;DR

Vevo tackles zero-shot voice imitation by disentangling linguistic content, timbre, and style through self-supervised, HuBERT-based representations. A content-style modeling stage (decoder-only autoregressive transformer) generates content-style tokens from content, guided by a style reference, while an acoustic modeling stage (flow-matching transformer) synthesizes Mel spectrograms conditioned on a timbre reference. The core innovation is the VQ-VAE tokenizer applied to HuBERT features, producing content-stage tokens $Q_c$ and content-style tokens $Q_s$ with carefully chosen vocabularies ($K_c$, $K_s$) to control information bottlenecks. Trained on 60K hours of audiobook data without style-specific fine-tuning, Vevo achieves competitive or superior results in accent and emotion conversion, zero-shot voice conversion, and text-to-speech tasks, demonstrating strong generalization, controllability, and scalability for diverse speech generation applications.

Abstract

The imitation of voice, targeted on specific speech attributes such as timbre and speaking style, is crucial in speech generation. However, existing methods rely heavily on annotated data, and struggle with effectively disentangling timbre and style, leading to challenges in achieving controllable generation, especially in zero-shot scenarios. To address these issues, we propose Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. Vevo operates in two core stages: (1) Content-Style Modeling: Given either text or speech's content tokens as input, we utilize an autoregressive transformer to generate the content-style tokens, which is prompted by a style reference; (2) Acoustic Modeling: Given the content-style tokens as input, we employ a flow-matching transformer to produce acoustic representations, which is prompted by a timbre reference. To obtain the content and content-style tokens of speech, we design a fully self-supervised approach that progressively decouples the timbre, style, and linguistic content of speech. Specifically, we adopt VQ-VAE as the tokenizer for the continuous hidden features of HuBERT. We treat the vocabulary size of the VQ-VAE codebook as the information bottleneck, and adjust it carefully to obtain the disentangled speech representations. Solely self-supervised trained on 60K hours of audiobook speech data, without any fine-tuning on style-specific corpora, Vevo matches or surpasses existing methods in accent and emotion conversion tasks. Additionally, Vevo's effectiveness in zero-shot voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility. Audio samples are available at https://versavoice.github.io.

Figures (5)

• Figure 1: Vevo inference pipeline. Notably, it can take either speech or text as input, and perform zero-shot imitation with controllable linguistic content (controlled by the source), style (controlled by the style reference), and timbre (controlled by the timbre reference) in a single forward pass.
• Figure 2: Content-style modeling based on autoregressive transformer. During inference, we employ both global style encoder and content-style tokenizer to enhance the effect of the style reference.
• Figure 3: Acoustic modeling based on a flow-matching transformer. During inference, we append the timbre reference to the rightmost (or leftmost) end, enabling timbre-controllable generation.
• Figure 4: Encoder and decoder architecture of our VQ-VAE tokenizer. $k$, $s$. $c_{in}$, and $c_{out}$ denote the kernel size, stride, input channels, and output channels. $h$ denotes the vocabulary size of tokenizer. $H$ denotes the hidden dimension of input representations (which is 1024 for HuBERT-Large). This figure is borrowed from the paper of RepCodec repcodec.
• Figure 5: Reference-global-guided continuation of $\mathcal{M}_{style}$ for inference.