Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VC-ENHANCE: Speech Restoration with Integrated Noise Suppression and Voice Conversion

Kyungguen Byun, Jason Filos, Erik Visser, Sunkuk Moon

TL;DR

This study proposes an explicit speech restoration method using a voice conversion (VC) technique for restoration after noise suppression, observing that high-quality speech can be restored through a diffusion-based voice conversion stage, conditioned on the target speaker embedding and speech content information extracted from the de-noised speech.

Abstract

Noise suppression (NS) algorithms are effective in improving speech quality in many cases. However, aggressive noise suppression can damage the target speech, reducing both speech intelligibility and quality despite removing the noise. This study proposes an explicit speech restoration method using a voice conversion (VC) technique for restoration after noise suppression. We observed that high-quality speech can be restored through a diffusion-based voice conversion stage, conditioned on the target speaker embedding and speech content information extracted from the de-noised speech. This speech restoration can achieve enhancement effects such as bandwidth extension, de-reverberation, and in-painting. Our experimental results demonstrate that this two-stage NS+VC framework outperforms single-stage enhancement models in terms of output speech quality, as measured by objective metrics, while scoring slightly lower in speech intelligibility. To further improve the intelligibility of the combined system, we propose a content encoder adaptation method for robust content extraction in noisy conditions.

VC-ENHANCE: Speech Restoration with Integrated Noise Suppression and Voice Conversion

TL;DR

This study proposes an explicit speech restoration method using a voice conversion (VC) technique for restoration after noise suppression, observing that high-quality speech can be restored through a diffusion-based voice conversion stage, conditioned on the target speaker embedding and speech content information extracted from the de-noised speech.

Abstract

Noise suppression (NS) algorithms are effective in improving speech quality in many cases. However, aggressive noise suppression can damage the target speech, reducing both speech intelligibility and quality despite removing the noise. This study proposes an explicit speech restoration method using a voice conversion (VC) technique for restoration after noise suppression. We observed that high-quality speech can be restored through a diffusion-based voice conversion stage, conditioned on the target speaker embedding and speech content information extracted from the de-noised speech. This speech restoration can achieve enhancement effects such as bandwidth extension, de-reverberation, and in-painting. Our experimental results demonstrate that this two-stage NS+VC framework outperforms single-stage enhancement models in terms of output speech quality, as measured by objective metrics, while scoring slightly lower in speech intelligibility. To further improve the intelligibility of the combined system, we propose a content encoder adaptation method for robust content extraction in noisy conditions.
Paper Structure (14 sections, 3 equations, 3 figures, 3 tables)

This paper contains 14 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion models
  4. HuBERT (Hidden-Unit BERT) model
  5. Method
  6. Noise suppression
  7. VC inspired speech restoration
  8. Content encoder adaptation
  9. Experiments
  10. Evaluation metrics & datasets
  11. Experimental result
  12. Ablation study
  13. Testing in various adverse conditions
  14. Conclusion

Figures (3)

  • Figure 1: Inference process of the proposed system.
  • Figure 2: Training and content encoder adaptation framework for voice conversion-based speech restoration model.
  • Figure 3: Input mixture including noise and packet-loss and associated outputs.