Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model

Jialong Zuo, Shengpeng Ji, Minghui Fang, Ziyue Jiang, Xize Cheng, Qian Yang, Wenrui Liu, Guangyan Zhang, Zehai Tu, Yiwen Guo, Zhou Zhao

TL;DR

The paper tackles expressiveness in voice conversion by moving beyond speaker identity preservation to richer prosody and emotion transfer. It introduces PFlow-VC, a conditional flow matching framework that conditions Mel synthesis on discrete pitch tokens from a self-supervised pitch VQVAE, along with target speaker prompts and time-varying timbre representations. Key contributions include a pitch quantization scheme with SMN-logf0, a dual-timbre encoder that combines global embeddings and time-varying timbre tokens, and an OT-CFM-based decoder that supports in-context pitch modeling and efficient generation. Experimental results on unseen LibriTTS and ESD data demonstrate superior zero-shot timbre transfer and emotion style consistency, with ablations validating the importance of pitch tokens and dynamic timbre representations for expressive VC.

Abstract

This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page https://speechai-demo.github.io/PFlow-VC/.

Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model

TL;DR

The paper tackles expressiveness in voice conversion by moving beyond speaker identity preservation to richer prosody and emotion transfer. It introduces PFlow-VC, a conditional flow matching framework that conditions Mel synthesis on discrete pitch tokens from a self-supervised pitch VQVAE, along with target speaker prompts and time-varying timbre representations. Key contributions include a pitch quantization scheme with SMN-logf0, a dual-timbre encoder that combines global embeddings and time-varying timbre tokens, and an OT-CFM-based decoder that supports in-context pitch modeling and efficient generation. Experimental results on unseen LibriTTS and ESD data demonstrate superior zero-shot timbre transfer and emotion style consistency, with ablations validating the importance of pitch tokens and dynamic timbre representations for expressive VC.

Abstract

This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page https://speechai-demo.github.io/PFlow-VC/.

Paper Structure

This paper contains 15 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. PFlow-VC Architecture
  3. Architecture Overview
  4. Pitch Quantizer
  5. Timbre Encoder
  6. Mask Conditional Flow Matching Model
  7. Trainging and Inference
  8. Experiments
  9. Experimental Setup
  10. Zero-shot VC Results
  11. Emotion Style Transfer Ability Evaluation
  12. Ablation Studies
  13. Pitch Modeling
  14. Speaker Representations
  15. Conclusions

Figures (2)

  • Figure 1: The overall architecture of PFlow-VC.
  • Figure 2: The visualization of emotion representations extracted from unseen ESD test set.