Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAIN-VC: Lightweight Speech Representation Disentanglement for One-shot Voice Conversion

Pengcheng Li, Jianzong Wang, Xulong Zhang, Yong Zhang, Jing Xiao, Ning Cheng

TL;DR

A model named MAIN-VC is proposed to effectively disentangle via a concise neural network to effectively disentangle via a concise neural network that utilizes Siamese encoders to learn clean representations, further enhanced by the designed mutual information estimator.

Abstract

One-shot voice conversion aims to change the timbre of any source speech to match that of the unseen target speaker with only one speech sample. Existing methods face difficulties in satisfactory speech representation disentanglement and suffer from sizable networks as some of them leverage numerous complex modules for disentanglement. In this paper, we propose a model named MAIN-VC to effectively disentangle via a concise neural network. The proposed model utilizes Siamese encoders to learn clean representations, further enhanced by the designed mutual information estimator. The Siamese structure and the newly designed convolution module contribute to the lightweight of our model while ensuring performance in diverse voice conversion tasks. The experimental results show that the proposed model achieves comparable subjective scores and exhibits improvements in objective metrics compared to existing methods in a one-shot voice conversion scenario.

MAIN-VC: Lightweight Speech Representation Disentanglement for One-shot Voice Conversion

TL;DR

A model named MAIN-VC is proposed to effectively disentangle via a concise neural network to effectively disentangle via a concise neural network that utilizes Siamese encoders to learn clean representations, further enhanced by the designed mutual information estimator.

Abstract

One-shot voice conversion aims to change the timbre of any source speech to match that of the unseen target speaker with only one speech sample. Existing methods face difficulties in satisfactory speech representation disentanglement and suffer from sizable networks as some of them leverage numerous complex modules for disentanglement. In this paper, we propose a model named MAIN-VC to effectively disentangle via a concise neural network. The proposed model utilizes Siamese encoders to learn clean representations, further enhanced by the designed mutual information estimator. The Siamese structure and the newly designed convolution module contribute to the lightweight of our model while ensuring performance in diverse voice conversion tasks. The experimental results show that the proposed model achieves comparable subjective scores and exhibits improvements in objective metrics compared to existing methods in a one-shot voice conversion scenario.
Paper Structure (18 sections, 10 equations, 4 figures, 3 tables)

This paper contains 18 sections, 10 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Speech Representation Disentanglement
  4. Neural Mutual Information Estimation
  5. Methodology
  6. Overall Architecure of MAIN-VC
  7. IN-based Speech Style Transfer
  8. Speaker Information Learning Module
  9. Mutual Information Enhanced Disentanglement
  10. Model Lightweighting
  11. Loss Function
  12. Experiments and Results
  13. Experimental Setup
  14. Metrics and Measurement
  15. Experimental Results
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Architecture of MAIN-VC and the training objectives.
  • Figure 2: The structure of the encoders and the details of APC.
  • Figure 3: The Mel-spectrogram samples of the one-shot VC task "Please call Stella".
  • Figure 4: The visualization of speaker representations extracted from 8 unseen speakers' utterances (4 males and 4 females, 100 utterances from each speaker).