Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAVITS: Exploring Prosody-aware VITS for End-to-End Emotional Voice Conversion

Tianhua Qi, Wenming Zheng, Cheng Lu, Yuan Zong, Hailun Lian

Abstract

In this paper, we propose Prosody-aware VITS (PAVITS) for emotional voice conversion (EVC), aiming to achieve two major objectives of EVC: high content naturalness and high emotional naturalness, which are crucial for meeting the demands of human perception. To improve the content naturalness of converted audio, we have developed an end-to-end EVC architecture inspired by the high audio quality of VITS. By seamlessly integrating an acoustic converter and vocoder, we effectively address the common issue of mismatch between emotional prosody training and run-time conversion that is prevalent in existing EVC models. To further enhance the emotional naturalness, we introduce an emotion descriptor to model the subtle prosody variations of different speech emotions. Additionally, we propose a prosody predictor, which predicts prosody features from text based on the provided emotion label. Notably, we introduce a prosody alignment loss to establish a connection between latent prosody features from two distinct modalities, ensuring effective training. Experimental results show that the performance of PAVITS is superior to the state-of-the-art EVC methods. Speech Samples are available at https://jeremychee4.github.io/pavits4EVC/ .

PAVITS: Exploring Prosody-aware VITS for End-to-End Emotional Voice Conversion

Abstract

In this paper, we propose Prosody-aware VITS (PAVITS) for emotional voice conversion (EVC), aiming to achieve two major objectives of EVC: high content naturalness and high emotional naturalness, which are crucial for meeting the demands of human perception. To improve the content naturalness of converted audio, we have developed an end-to-end EVC architecture inspired by the high audio quality of VITS. By seamlessly integrating an acoustic converter and vocoder, we effectively address the common issue of mismatch between emotional prosody training and run-time conversion that is prevalent in existing EVC models. To further enhance the emotional naturalness, we introduce an emotion descriptor to model the subtle prosody variations of different speech emotions. Additionally, we propose a prosody predictor, which predicts prosody features from text based on the provided emotion label. Notably, we introduce a prosody alignment loss to establish a connection between latent prosody features from two distinct modalities, ensuring effective training. Experimental results show that the performance of PAVITS is superior to the state-of-the-art EVC methods. Speech Samples are available at https://jeremychee4.github.io/pavits4EVC/ .
Paper Structure (14 sections, 10 equations, 3 figures, 3 tables)

This paper contains 14 sections, 10 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Textual prosody prediction module
  4. Acoustic prosody modeling module
  5. Information alignment module
  6. Emotional speech synthesis module
  7. Final loss
  8. Run-time conversion
  9. Experiments
  10. Dataset
  11. Experimental Setup
  12. Results & Discussion
  13. Ablation Study
  14. Conclusion

Figures (3)

  • Figure 1: Architecture of PAVITS.
  • Figure 2: Emotional similarity test with 95% confidence interval following zhou2021limited.
  • Figure 3: Spectrogram of a testing clip (happy), from top to bottom are ground truth, converted by original VITS, and proposed PAVITS.