Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts

Tianhua Qi, Shiyan Wang, Cheng Lu, Tengfei Song, Hao Yang, Zhanglin Wu, Wenming Zheng

TL;DR

PromptEVC tackles controllable emotional voice conversion by replacing predefined labels or reference-only controls with natural language prompts. The method introduces an emotion descriptor (RoBERTa-based) to generate e_txt and a diffusion-driven prompt mapper to produce a fine-grained embedding e_pm conditioned on e_txt and a reference embedding e_ref, yielding a robust emotion representation h_emo. Built on a CVAE framework, PromptEVC incorporates a HuBERT-based prosody module with a duration regulator and a prosody predictor to align rhythm and pitch with linguistic content and emotional cues, while a speaker encoder with an F0 constraint preserves identity. The waveform is reconstructed via a posterior encoder and a reversible flow, with adversarial training to boost naturalness, and the final loss combines L_sp, L_f, L_adv, L_pm, L_rhy, and L_spk. Experiments on TextrolSpeech show PromptEVC achieves strong controllability and diversity, outperforming several baselines in objective metrics (MCD, RMSE_F0, CER) and subjective MOS, with ablations confirming the importance of each component and the effectiveness of natural language prompts for emotion manipulation.

Abstract

Controllable emotional voice conversion (EVC) aims to manipulate emotional expressions to increase the diversity of synthesized speech. Existing methods typically rely on predefined labels, reference audios, or prespecified factor values, often overlooking individual differences in emotion perception and expression. In this paper, we introduce PromptEVC that utilizes natural language prompts for precise and flexible emotion control. To bridge text descriptions with emotional speech, we propose emotion descriptor and prompt mapper to generate fine-grained emotion embeddings, trained jointly with reference embeddings. To enhance naturalness, we present a prosody modeling and control pipeline that adjusts the rhythm based on linguistic content and emotional cues. Additionally, a speaker encoder is incorporated to preserve identity. Experimental results demonstrate that PromptEVC outperforms state-of-the-art controllable EVC methods in emotion conversion, intensity control, mixed emotion synthesis, and prosody manipulation. Speech samples are available at https://jeremychee4.github.io/PromptEVC/.

PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts

TL;DR

PromptEVC tackles controllable emotional voice conversion by replacing predefined labels or reference-only controls with natural language prompts. The method introduces an emotion descriptor (RoBERTa-based) to generate e_txt and a diffusion-driven prompt mapper to produce a fine-grained embedding e_pm conditioned on e_txt and a reference embedding e_ref, yielding a robust emotion representation h_emo. Built on a CVAE framework, PromptEVC incorporates a HuBERT-based prosody module with a duration regulator and a prosody predictor to align rhythm and pitch with linguistic content and emotional cues, while a speaker encoder with an F0 constraint preserves identity. The waveform is reconstructed via a posterior encoder and a reversible flow, with adversarial training to boost naturalness, and the final loss combines L_sp, L_f, L_adv, L_pm, L_rhy, and L_spk. Experiments on TextrolSpeech show PromptEVC achieves strong controllability and diversity, outperforming several baselines in objective metrics (MCD, RMSE_F0, CER) and subjective MOS, with ablations confirming the importance of each component and the effectiveness of natural language prompts for emotion manipulation.

Abstract

Controllable emotional voice conversion (EVC) aims to manipulate emotional expressions to increase the diversity of synthesized speech. Existing methods typically rely on predefined labels, reference audios, or prespecified factor values, often overlooking individual differences in emotion perception and expression. In this paper, we introduce PromptEVC that utilizes natural language prompts for precise and flexible emotion control. To bridge text descriptions with emotional speech, we propose emotion descriptor and prompt mapper to generate fine-grained emotion embeddings, trained jointly with reference embeddings. To enhance naturalness, we present a prosody modeling and control pipeline that adjusts the rhythm based on linguistic content and emotional cues. Additionally, a speaker encoder is incorporated to preserve identity. Experimental results demonstrate that PromptEVC outperforms state-of-the-art controllable EVC methods in emotion conversion, intensity control, mixed emotion synthesis, and prosody manipulation. Speech samples are available at https://jeremychee4.github.io/PromptEVC/.

Paper Structure

This paper contains 14 sections, 7 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Text-emotion mapping
  4. Prosody modeling and control
  5. Speaker identity preservation
  6. Waveform reconstruction
  7. Final loss
  8. Experiments
  9. Experimental setup
  10. Model performance
  11. Ablation study
  12. Controllability of natural language prompts
  13. Conclusion
  14. Acknowledgment

Figures (2)

  • Figure 1: Diagram of proposed PromptEVC, depicting the overall architecture (a) and the prosody modeling and control module (b).
  • Figure 2: Mel-spectrograms and pitch contours of the converted audios for different natural language prompts: "Convert this neutral utterance into a happy/surprise tone" and "Inject some happiness into this sentence with a hint of surprise."