Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed

Meiying Chen, Zhiyao Duan

TL;DR

ControlVC tackles the absence of time-varying para-linguistic control in voice conversion by introducing a three-stage pipeline that applies TD-PSOLA speed modulation, pitch contour adjustment, and a neural vocoder conditioned on fused embeddings. It leverages pre-trained encoders for pitch (VQ-VAE), linguistic content (HuBERT), and speaker timbre (speaker encoder), fused with a HiFi-GAN vocoder to produce controllable, zero-shot conversions. Objective and subjective evaluations on non-parallel VC show that ControlVC outperforms two baselines in speech quality and intelligibility while achieving reliable per-frame pitch and speed control. This work enables more expressive and practically deployable VC systems by providing interpretable, time-varying controls for para-linguistic features.

Abstract

Recent developments in neural speech synthesis and vocoding have sparked a renewed interest in voice conversion (VC). Beyond timbre transfer, achieving controllability on para-linguistic parameters such as pitch and Speed is critical in deploying VC systems in many application scenarios. Existing studies, however, either only provide utterance-level global control or lack interpretability on the controls. In this paper, we propose ControlVC, the first neural voice conversion system that achieves time-varying controls on pitch and speed. ControlVC uses pre-trained encoders to compute pitch and linguistic embeddings from the source utterance and speaker embeddings from the target utterance. These embeddings are then concatenated and converted to speech using a vocoder. It achieves speed control through TD-PSOLA pre-processing on the source utterance, and achieves pitch control by manipulating the pitch contour before feeding it to the pitch encoder. Systematic subjective and objective evaluations are conducted to assess the speech quality and controllability. Results show that, on non-parallel and zero-shot conversion tasks, ControlVC significantly outperforms two other self-constructed baselines on speech quality, and it can successfully achieve time-varying pitch and speed control.

ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed

TL;DR

ControlVC tackles the absence of time-varying para-linguistic control in voice conversion by introducing a three-stage pipeline that applies TD-PSOLA speed modulation, pitch contour adjustment, and a neural vocoder conditioned on fused embeddings. It leverages pre-trained encoders for pitch (VQ-VAE), linguistic content (HuBERT), and speaker timbre (speaker encoder), fused with a HiFi-GAN vocoder to produce controllable, zero-shot conversions. Objective and subjective evaluations on non-parallel VC show that ControlVC outperforms two baselines in speech quality and intelligibility while achieving reliable per-frame pitch and speed control. This work enables more expressive and practically deployable VC systems by providing interpretable, time-varying controls for para-linguistic features.

Abstract

Recent developments in neural speech synthesis and vocoding have sparked a renewed interest in voice conversion (VC). Beyond timbre transfer, achieving controllability on para-linguistic parameters such as pitch and Speed is critical in deploying VC systems in many application scenarios. Existing studies, however, either only provide utterance-level global control or lack interpretability on the controls. In this paper, we propose ControlVC, the first neural voice conversion system that achieves time-varying controls on pitch and speed. ControlVC uses pre-trained encoders to compute pitch and linguistic embeddings from the source utterance and speaker embeddings from the target utterance. These embeddings are then concatenated and converted to speech using a vocoder. It achieves speed control through TD-PSOLA pre-processing on the source utterance, and achieves pitch control by manipulating the pitch contour before feeding it to the pitch encoder. Systematic subjective and objective evaluations are conducted to assess the speech quality and controllability. Results show that, on non-parallel and zero-shot conversion tasks, ControlVC significantly outperforms two other self-constructed baselines on speech quality, and it can successfully achieve time-varying pitch and speed control.
Paper Structure (16 sections, 2 equations, 2 figures, 2 tables)

This paper contains 16 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed ControlVC System
  3. Overview
  4. TD-PSOLA Prepossessing and Speed Control
  5. Pitch Control and Pitch Embedding
  6. Linguistic Embedding
  7. Speaker Embedding
  8. HiFi-GAN Neural Vocoder
  9. Experiments
  10. Dataset
  11. Baseline Methods
  12. Training
  13. Experimental Setup
  14. Objective Evaluation
  15. Subjective Evaluation
  16. ...and 1 more sections

Figures (2)

  • Figure 1: System Overview.
  • Figure 2: MOS results on audio quality (naturalness and timbre similarity) with 95% confidence intervals.