Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VCTR: A Transformer-Based Model for Non-parallel Voice Conversion

Maharnab Saikia

TL;DR

VCTR addresses non-parallel voice conversion by introducing a transformer-based generator that combines local and global feature modeling via Hybrid Perception Blocks ($HPB$) and Dual Pruned Self-Attention ($DPSA$), trained with a contrastive adversarial objective that includes PatchNCE losses. The approach uses a ResNet backbone, nine HPBs at the bottleneck, and a PatchGAN discriminator to handle mel-spectrogram inputs treated as 2D images, achieving strong speaker similarity while maintaining a reasonable computational footprint. Key findings show VCTR achieving comparable or better voice similarity than strong non-parallel baselines (VAE, CycleGAN-VC, CVC, CNEG-VC) on the VCTK dataset, with ablations confirming the importance of DPSA, local perception, and L2 normalization in HPB. Limitations include persistent mode collapse in many-to-one scenarios and fixed-length input/output constraints, motivating future work on spectral diversity and adaptive-length processing for broader applicability.

Abstract

Non-parallel voice conversion aims to convert voice from a source domain to a target domain without paired training data. Cycle-Consistent Generative Adversarial Networks (CycleGAN) and Variational Autoencoders (VAE) have been used for this task, but these models suffer from difficult training and unsatisfactory results. Later, Contrastive Voice Conversion (CVC) was introduced, utilizing a contrastive learning-based approach to address these issues. However, these methods use CNN-based generators, which can capture local semantics but lacks the ability to capture long-range dependencies necessary for global semantics. In this paper, we propose VCTR, an efficient method for non-parallel voice conversion that leverages the Hybrid Perception Block (HPB) and Dual Pruned Self-Attention (DPSA) along with a contrastive learning-based adversarial approach. The code can be found in https://github.com/Maharnab-Saikia/VCTR.

VCTR: A Transformer-Based Model for Non-parallel Voice Conversion

TL;DR

VCTR addresses non-parallel voice conversion by introducing a transformer-based generator that combines local and global feature modeling via Hybrid Perception Blocks () and Dual Pruned Self-Attention (), trained with a contrastive adversarial objective that includes PatchNCE losses. The approach uses a ResNet backbone, nine HPBs at the bottleneck, and a PatchGAN discriminator to handle mel-spectrogram inputs treated as 2D images, achieving strong speaker similarity while maintaining a reasonable computational footprint. Key findings show VCTR achieving comparable or better voice similarity than strong non-parallel baselines (VAE, CycleGAN-VC, CVC, CNEG-VC) on the VCTK dataset, with ablations confirming the importance of DPSA, local perception, and L2 normalization in HPB. Limitations include persistent mode collapse in many-to-one scenarios and fixed-length input/output constraints, motivating future work on spectral diversity and adaptive-length processing for broader applicability.

Abstract

Non-parallel voice conversion aims to convert voice from a source domain to a target domain without paired training data. Cycle-Consistent Generative Adversarial Networks (CycleGAN) and Variational Autoencoders (VAE) have been used for this task, but these models suffer from difficult training and unsatisfactory results. Later, Contrastive Voice Conversion (CVC) was introduced, utilizing a contrastive learning-based approach to address these issues. However, these methods use CNN-based generators, which can capture local semantics but lacks the ability to capture long-range dependencies necessary for global semantics. In this paper, we propose VCTR, an efficient method for non-parallel voice conversion that leverages the Hybrid Perception Block (HPB) and Dual Pruned Self-Attention (DPSA) along with a contrastive learning-based adversarial approach. The code can be found in https://github.com/Maharnab-Saikia/VCTR.
Paper Structure (13 sections, 10 equations, 3 figures, 3 tables)

This paper contains 13 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contrastive Learning for Unpaired Translation
  4. Vision Transformer
  5. Generative Adversarial Networks
  6. Methods
  7. Model Architecture
  8. Discriminator
  9. Loss Functions
  10. Experiments
  11. Non-Parallel Voice Conversion
  12. Ablation Study
  13. Conclusion

Figures (3)

  • Figure 1: The VCTR architecture, consists of a ResNet on the left and a single Hybrid Perception Block (HPB) on the middle. Concatenation is denoted by 'C,' while 'IN' represents Instance Normalization and 'Conv' denotes Convolution. Dual Pruned Self-Attention (DPSA) on the right, 'L2 Norm' to indicate L2 normalization and 'X' to denote matrix multiplication. The dotted box represents operations within a single attention head. For best clarity, view the diagram zoomed in.
  • Figure 2: Patchwise contrastive learning is used for one-sided translation, A generated output patch should be closer to its corresponding input patch compared to other random patches. To achieve this, we use a multi-layer patch-wise contrastive loss, which maximizes mutual information between corresponding input and output patches.
  • Figure 3: The loss curve from generator training is shown, with CVC represented in royal blue and our model, VCTR, in crimson. The graph shows loss spikes during male-to-female (one-to-one) voice conversion training.