Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Vocoders as Speech Enhancers

Andong Li, Zhihang Sun, Fengyuan Hao, Xiaodong Li, Chengshi Zheng

TL;DR

This work reframes speech enhancement (SE) and neural vocoding as a unified speech restoration problem through spectral rank manipulation. It shows that vocoding requires increasing spectral rank, while denoising tends to decrease rank, and that SE models can be repurposed for vocoding tasks; a joint training regime enables a single model to perform both tasks with performance close to dedicated models. The approach is evaluated on LJSpeech and LibriTTS+Noise with both TF-domain and time-domain networks, demonstrating competitive vocoding results and effective dual-task performance under a shared training objective. The findings suggest a practical path toward a unified, rank-aware framework for restoring speech, with potential benefits for efficiency and versatility in real-world applications.

Abstract

Speech enhancement (SE) and neural vocoding are traditionally viewed as separate tasks. In this work, we observe them under a common thread: the rank behavior of these processes. This observation prompts two key questions: \textit{Can a model designed for one task's rank degradation be adapted for the other?} and \textit{Is it possible to address both tasks using a unified model?} Our empirical findings demonstrate that existing speech enhancement models can be successfully trained to perform vocoding tasks, and a single model, when jointly trained, can effectively handle both tasks with performance comparable to separately trained models. These results suggest that speech enhancement and neural vocoding can be unified under a broader framework of speech restoration. Code: https://github.com/Andong-Li-speech/Neural-Vocoders-as-Speech-Enhancers.

Neural Vocoders as Speech Enhancers

TL;DR

This work reframes speech enhancement (SE) and neural vocoding as a unified speech restoration problem through spectral rank manipulation. It shows that vocoding requires increasing spectral rank, while denoising tends to decrease rank, and that SE models can be repurposed for vocoding tasks; a joint training regime enables a single model to perform both tasks with performance close to dedicated models. The approach is evaluated on LJSpeech and LibriTTS+Noise with both TF-domain and time-domain networks, demonstrating competitive vocoding results and effective dual-task performance under a shared training objective. The findings suggest a practical path toward a unified, rank-aware framework for restoring speech, with potential benefits for efficiency and versatility in real-world applications.

Abstract

Speech enhancement (SE) and neural vocoding are traditionally viewed as separate tasks. In this work, we observe them under a common thread: the rank behavior of these processes. This observation prompts two key questions: \textit{Can a model designed for one task's rank degradation be adapted for the other?} and \textit{Is it possible to address both tasks using a unified model?} Our empirical findings demonstrate that existing speech enhancement models can be successfully trained to perform vocoding tasks, and a single model, when jointly trained, can effectively handle both tasks with performance comparable to separately trained models. These results suggest that speech enhancement and neural vocoding can be unified under a broader framework of speech restoration. Code: https://github.com/Andong-Li-speech/Neural-Vocoders-as-Speech-Enhancers.
Paper Structure (16 sections, 9 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 16 sections, 9 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Signal Models and Rank Analysis
  3. Learning Commonalities between SE and Vocoding
  4. T-F-domain and time-domain SE networks
  5. Joint denoising-vocoding training
  6. Experiments
  7. Dataset
  8. LJSpeech
  9. LibriTTS+Noise
  10. Models and training configurations
  11. Results and Analysis
  12. Ablation studies
  13. Performance Comparisons for speech vocoding task
  14. Performance of joint denoising and vocoding training
  15. Conclusions
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Illustrations of the signal degradation process w.r.t. denoising and vocoding tasks.
  • Figure 2: Relative rank difference w.r.t. target spectrum of denoising and vocoding tasks. The ranks are calculated from the training set of Voicebank-Demand benchmark veaux2013voice. The absolute threshold $\eta$ is set to 0.5 for rank calculation and better visualization.
  • Figure 3: Metric comparisons for joint denoising-vocoding task and its single task versions. For denoising task, WB-PESQ and DNSMOS P.835 reddy2022dnsmos are adopted, and WB-PESQ and UTMOS for vocoding.