Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion

TL;DR

This work tackles the naturalness gap in GAN-based non-parallel voice conversion by introducing CLOT-GAN-VC, a single-generator framework guided by multiple discriminators (DCNN, ViT, Conformer) within a collective learning mechanism. It integrates an optimal transport loss via the Sinkhorn distance to align source and target mel-spectrogram distributions, with cycle-consistency and identity losses to stabilize learning. Empirical results across VCC 2018, VCTK, and CMU Arctic show improvements in objective metrics (MCD, MSD) and subjective MOS over state-of-the-art baselines, with ablation studies underscoring the contributions of OT loss and multi-discriminator collaboration. The approach holds promise for robust VC in low-resource settings and broader non-parallel VC tasks.

Abstract

After demonstrating significant success in image synthesis, Generative Adversarial Network (GAN) models have likewise made significant progress in the field of speech synthesis, leveraging their capacity to adapt the precise distribution of target data through adversarial learning processes. Notably, in the realm of State-Of-The-Art (SOTA) GAN-based Voice Conversion (VC) models, there exists a substantial disparity in naturalness between real and GAN-generated speech samples. Furthermore, while many GAN models currently operate on a single generator discriminator learning approach, optimizing target data distribution is more effectively achievable through a single generator multi-discriminator learning scheme. Hence, this study introduces a novel GAN model named Collective Learning Mechanism-based Optimal Transport GAN (CLOT-GAN) model, incorporating multiple discriminators, including the Deep Convolutional Neural Network (DCNN) model, Vision Transformer (ViT), and conformer. The objective of integrating various discriminators lies in their ability to comprehend the formant distribution of mel-spectrograms, facilitated by a collective learning mechanism. Simultaneously, the inclusion of Optimal Transport (OT) loss aims to precisely bridge the gap between the source and target data distribution, employing the principles of OT theory. The experimental validation on VCC 2018, VCTK, and CMU-Arctic datasets confirms that the CLOT-GAN-VC model outperforms existing VC models in objective and subjective assessments.

Figures (2)

• Figure 1: Overview of the proposed CLOT-GAN-VC model
• Figure 2: Mel-spectrograms (M to F VC), generated using CLOT-GAN-VC, MaskCycleGAN-VC, and MelGAN-VC, along with Grad-CAM-based visualization for each discriminator setting