Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy

Linhan Ma, Xinfa Zhu, Yuanjun Lv, Zhichao Wang, Ziqian Wang, Wendi He, Hongbin Zhou, Lei Xie

TL;DR

Vec-Tok-VC+ tackles zero-shot voice conversion by decoupling content and timbre with a residual-enhanced K-Means decoupler and a prompt-based conformer converter that uses a 3s target speaker prompt. It introduces a dual-mode training scheme with teacher-guided refinement to reduce training-inference mismatch and a multi-codebook progressive loss to guide layer-wise representation learning from coarse to fine. Experiments on large multilingual datasets show improvements in naturalness, intelligibility, and speaker similarity over strong baselines, including cross-lingual scenarios and noisy conditions. The method enables robust, data-efficient zero-shot VC with practical deployment potential due to its short prompts and improved content fidelity.

Abstract

Zero-shot voice conversion (VC) aims to transform source speech into arbitrary unseen target voice while keeping the linguistic content unchanged. Recent VC methods have made significant progress, but semantic losses in the decoupling process as well as training-inference mismatch still hinder conversion performance. In this paper, we propose Vec-Tok-VC+, a novel prompt-based zero-shot VC model improved from Vec-Tok Codec, achieving voice conversion given only a 3s target speaker prompt. We design a residual-enhanced K-Means decoupler to enhance the semantic content extraction with a two-layer clustering process. Besides, we employ teacher-guided refinement to simulate the conversion process to eliminate the training-inference mismatch, forming a dual-mode training strategy. Furthermore, we design a multi-codebook progressive loss function to constrain the layer-wise output of the model from coarse to fine to improve speaker similarity and content accuracy. Objective and subjective evaluations demonstrate that Vec-Tok-VC+ outperforms the strong baselines in naturalness, intelligibility, and speaker similarity.

Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy

TL;DR

Vec-Tok-VC+ tackles zero-shot voice conversion by decoupling content and timbre with a residual-enhanced K-Means decoupler and a prompt-based conformer converter that uses a 3s target speaker prompt. It introduces a dual-mode training scheme with teacher-guided refinement to reduce training-inference mismatch and a multi-codebook progressive loss to guide layer-wise representation learning from coarse to fine. Experiments on large multilingual datasets show improvements in naturalness, intelligibility, and speaker similarity over strong baselines, including cross-lingual scenarios and noisy conditions. The method enables robust, data-efficient zero-shot VC with practical deployment potential due to its short prompts and improved content fidelity.

Abstract

Zero-shot voice conversion (VC) aims to transform source speech into arbitrary unseen target voice while keeping the linguistic content unchanged. Recent VC methods have made significant progress, but semantic losses in the decoupling process as well as training-inference mismatch still hinder conversion performance. In this paper, we propose Vec-Tok-VC+, a novel prompt-based zero-shot VC model improved from Vec-Tok Codec, achieving voice conversion given only a 3s target speaker prompt. We design a residual-enhanced K-Means decoupler to enhance the semantic content extraction with a two-layer clustering process. Besides, we employ teacher-guided refinement to simulate the conversion process to eliminate the training-inference mismatch, forming a dual-mode training strategy. Furthermore, we design a multi-codebook progressive loss function to constrain the layer-wise output of the model from coarse to fine to improve speaker similarity and content accuracy. Objective and subjective evaluations demonstrate that Vec-Tok-VC+ outperforms the strong baselines in naturalness, intelligibility, and speaker similarity.
Paper Structure (11 sections, 2 figures, 2 tables)

This paper contains 11 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed approach
  3. System overview
  4. Residual-enhanced K-Means decoupler
  5. Dual-mode training with teacher-guided refinement
  6. Prompt-based conformer converter
  7. Experiments
  8. Experimental setup
  9. Zero-shot voice conversion results
  10. Ablation study
  11. Conclusion

Figures (2)

  • Figure 1: The overview of Vec-Tok-VC+.
  • Figure 2: The details of Vec-Tok-VC+. (a): the residual-enhanced K-Means decoupler. (b): the dual-mode teacher guidance module. (c): the converter and multi-codebook progressive constraint.