Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Voice-ENHANCE: Speech Restoration using a Diffusion-based Voice Conversion Framework

Kyungguen Byun, Jason Filos, Erik Visser, Sunkuk Moon

TL;DR

This paper tackles speech restoration under noisy conditions by proposing a two-stage pipeline that first applies a speaker-agnostic Generative Speech Restoration (GSR) front end and then a diffusion-based Voice Conversion (VC) back end guided by clean speaker embeddings. The GSR handles multiple distortions via a ResU-Net and a HiFi-GAN vocoder, while the VC-inspired stage leverages HuBERT+VQ content representations and an ECAPA-TDNN speaker embedding to condition a diffusion decoder, with classifier-free guidance and a balanced loss $L_{total} = L_{d} + \alpha L_{enc}$ ($\alpha=0.5$). Evaluations on VCTK-DEMAND and UNIVERSE show competitive to SOTA performance across non-intrusive metrics like NISQA, UTMOS, WV-MOS, and DNSMOS, with a relatively small to moderate model size compared to large baselines. The approach offers robust perceptual quality in real-world noisy environments and reduces dependence on transcripts, highlighting its practical utility and potential for end-to-end training enhancements in the future.

Abstract

We propose a speech enhancement system that combines speaker-agnostic speech restoration with voice conversion (VC) to obtain a studio-level quality speech signal. While voice conversion models are typically used to change speaker characteristics, they can also serve as a means of speech restoration when the target speaker is the same as the source speaker. However, since VC models are vulnerable to noisy conditions, we have included a generative speech restoration (GSR) model at the front end of our proposed system. The GSR model performs noise suppression and restores speech damage incurred during that process without knowledge about the target speaker. The VC stage then uses guidance from clean speaker embeddings to further restore the output speech. By employing this two-stage approach, we have achieved speech quality objective metric scores comparable to state-of-the-art (SOTA) methods across multiple datasets.

Voice-ENHANCE: Speech Restoration using a Diffusion-based Voice Conversion Framework

TL;DR

This paper tackles speech restoration under noisy conditions by proposing a two-stage pipeline that first applies a speaker-agnostic Generative Speech Restoration (GSR) front end and then a diffusion-based Voice Conversion (VC) back end guided by clean speaker embeddings. The GSR handles multiple distortions via a ResU-Net and a HiFi-GAN vocoder, while the VC-inspired stage leverages HuBERT+VQ content representations and an ECAPA-TDNN speaker embedding to condition a diffusion decoder, with classifier-free guidance and a balanced loss (). Evaluations on VCTK-DEMAND and UNIVERSE show competitive to SOTA performance across non-intrusive metrics like NISQA, UTMOS, WV-MOS, and DNSMOS, with a relatively small to moderate model size compared to large baselines. The approach offers robust perceptual quality in real-world noisy environments and reduces dependence on transcripts, highlighting its practical utility and potential for end-to-end training enhancements in the future.

Abstract

We propose a speech enhancement system that combines speaker-agnostic speech restoration with voice conversion (VC) to obtain a studio-level quality speech signal. While voice conversion models are typically used to change speaker characteristics, they can also serve as a means of speech restoration when the target speaker is the same as the source speaker. However, since VC models are vulnerable to noisy conditions, we have included a generative speech restoration (GSR) model at the front end of our proposed system. The GSR model performs noise suppression and restores speech damage incurred during that process without knowledge about the target speaker. The VC stage then uses guidance from clean speaker embeddings to further restore the output speech. By employing this two-stage approach, we have achieved speech quality objective metric scores comparable to state-of-the-art (SOTA) methods across multiple datasets.

Paper Structure

This paper contains 10 sections, 3 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Proposed method
  4. Generative Speech Restoration (GSR)
  5. VC-Inspired Speech Restoration
  6. Evaluation
  7. Database and Training Setting
  8. Comparison with Other Methods
  9. Discussion and Ablation Study
  10. Conclusion

Figures (3)

  • Figure 1: Overview of Voice-Enhance framework, noisy mixture is given to speaker-agnostic Generative speech restoration (GSR) module, then VC-inspired generative model generates high quality output guided by speaker embedding from the clean speech enrollment.
  • Figure 2: Training and content encoder adaptation framework for voice conversion-based speech restoration model.
  • Figure 3: Spectrogram comparison: (a) input mixture with noise and packet loss, (b) GSR output, and (c) final output from the proposed GSR+VC model.