OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching

TL;DR

OZSpeech tackles zero-shot TTS by reformulating diffusion-based flow matching into a one-step process that starts from a learned prior rather than Gaussian noise. It jointly learns a Prior Codes Generator and a Vector Field Estimator to map six discretized code sequences, produced from phonemes via FACodec, toward the target speech using a streamlined OT-CFM objective with a single sampling step. The system emphasizes disentangled token-based representations for content, prosody, and acoustic attributes, achieving strong WER improvements and robust prosody transfer while maintaining a small footprint and faster inference compared to baselines. These contributions enable efficient, high-fidelity cloning of unseen speakers from short prompts, with demonstrated robustness to noisy prompts and potential for multilingual extension.

Abstract

Text-to-speech (TTS) systems have seen significant advancements in recent years, driven by improvements in deep learning and neural network architectures. Viewing the output speech as a data distribution, previous approaches often employ traditional speech representations, such as waveforms or spectrograms, within the Flow Matching framework. However, these methods have limitations, including overlooking various speech attributes and incurring high computational costs due to additional constraints introduced during training. To address these challenges, we introduce OZSpeech, the first TTS method to explore optimal transport conditional flow matching with one-step sampling and a learned prior as the condition, effectively disregarding preceding states and reducing the number of sampling steps. Our approach operates on disentangled, factorized components of speech in token format, enabling accurate modeling of each speech attribute, which enhances the TTS system's ability to precisely clone the prompt speech. Experimental results show that our method achieves promising performance over existing methods in content accuracy, naturalness, prosody generation, and speaker style preservation. Audio samples are available at our demo page https://ozspeech.github.io/OZSpeech_Web/.

Figures (2)

• Figure 1: Overview of OZSpeech: (a) The overall architecture: The text prompt is converted to phonemes and then into prior codes via the Prior Codes Generator. Simultaneously, the audio prompt is encoded into codes using the FACodec Encoder. These codes are concatenated along the sequence dimension and fed into the OT-CFM Vector Field Estimator, which generates codes preserving the text content and acoustic attributes. Finally, the FACodec Decoder converts them into output speech. (b) The Prior Codes Generator $f_\psi(\cdot)$ produces sequences of phoneme-aligned codes. (c) The Vector Field Estimator refines these codes with the prosody and acoustic details from the acoustic prompt. Before being fed through $v_\theta(\cdot, \cdot)$, six sequences of codes are first enhanced via Quantizer Embedding, which serves as an identifier for each sequence within the hidden space. These embeddings are then folded along the hidden dimension and processed by the network to estimate the velocity of the prior codes.
• Figure 2: Boxplots showing the distributions of performance metrics (WER, UTMOS, and SIM-O) on the LibriSpeech test-clean dataset for each model, evaluated across different audio prompt lengths.