Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model

Zongyang Du, Junchen Lu, Kun Zhou, Lakshmish Kaushik, Berrak Sisman

TL;DR

This paper addresses expressive voice conversion for emotional speakers and the limitations of vocoder-reliant approaches. It introduces DEVC, a fully end-to-end expressive VC framework that uses a conditional diffusion model with content conditioning from soft speech units and deep features from speech emotion recognition and speaker verification. The authors show that speaker embeddings learned from neutral speech contain speaker-dependent emotional cues and that combining speaker-dependent and speaker-independent emotional representations enables any-to-any conversion. Experiments on the ESD dataset demonstrate improvements in objective and subjective metrics, and the method operates without a vocoder, enabling flexible, high-quality expressive VC for seen and unseen speakers.

Abstract

Expressive voice conversion (VC) conducts speaker identity conversion for emotional speakers by jointly converting speaker identity and emotional style. Emotional style modeling for arbitrary speakers in expressive VC has not been extensively explored. Previous approaches have relied on vocoders for speech reconstruction, which makes speech quality heavily dependent on the performance of vocoders. A major challenge of expressive VC lies in emotion prosody modeling. To address these challenges, this paper proposes a fully end-to-end expressive VC framework based on a conditional denoising diffusion probabilistic model (DDPM). We utilize speech units derived from self-supervised speech models as content conditioning, along with deep features extracted from speech emotion recognition and speaker verification systems to model emotional style and speaker identity. Objective and subjective evaluations show the effectiveness of our framework. Codes and samples are publicly available.

Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model

TL;DR

This paper addresses expressive voice conversion for emotional speakers and the limitations of vocoder-reliant approaches. It introduces DEVC, a fully end-to-end expressive VC framework that uses a conditional diffusion model with content conditioning from soft speech units and deep features from speech emotion recognition and speaker verification. The authors show that speaker embeddings learned from neutral speech contain speaker-dependent emotional cues and that combining speaker-dependent and speaker-independent emotional representations enables any-to-any conversion. Experiments on the ESD dataset demonstrate improvements in objective and subjective metrics, and the method operates without a vocoder, enabling flexible, high-quality expressive VC for seen and unseen speakers.

Abstract

Expressive voice conversion (VC) conducts speaker identity conversion for emotional speakers by jointly converting speaker identity and emotional style. Emotional style modeling for arbitrary speakers in expressive VC has not been extensively explored. Previous approaches have relied on vocoders for speech reconstruction, which makes speech quality heavily dependent on the performance of vocoders. A major challenge of expressive VC lies in emotion prosody modeling. To address these challenges, this paper proposes a fully end-to-end expressive VC framework based on a conditional denoising diffusion probabilistic model (DDPM). We utilize speech units derived from self-supervised speech models as content conditioning, along with deep features extracted from speech emotion recognition and speaker verification systems to model emotional style and speaker identity. Objective and subjective evaluations show the effectiveness of our framework. Codes and samples are publicly available.
Paper Structure (18 sections, 5 equations, 5 figures, 4 tables)

This paper contains 18 sections, 5 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Expressive Voice Conversion
  4. Denoising Diffusion Probabilistic Models in Speech Synthesis
  5. Representation Learning with Self-Supervised Models
  6. Emotion and Speaker Representations: A Novel Analysis for Expressive VC
  7. Proposed Method - DEVC
  8. Training Stage
  9. Run-time Conversion
  10. Experiments
  11. Dataset
  12. Experimental Setup
  13. Baselines
  14. Objective Evaluations
  15. Subjective Evaluation
  16. ...and 3 more sections

Figures (5)

  • Figure 1: A comparison between the conversion of different speech components across VC9262021, expressive VCdu2021expressive, and emotional VCzhou2021emotional.
  • Figure 2: Visualization of speaker representations and speaker-independent emotion representations of 50 randomly selected utterances from 4 speakers across 4 different emotional states. Each point represents one expressive utterance and the legend indicates speaker identity and emotional state information.
  • Figure 3: An illustration of the training phase of the proposed DEVC, where the green boxes represent the modules that are involved in the training while the others are not.
  • Figure 4: An illustration of the run-time phase of the proposed DEVC, highlighting that waveform generation occurs without the use of a vocoder.
  • Figure 5: ABX preference results for S2S, S2U, and U2S settings to evaluate: (a) speaker similarity and (b) emotional style similarity. We used Baseline (JES-StarGAN) for S2S, and Baseline-U for S2U and U2S settings.