VoiceFixer: Toward General Speech Restoration with Neural Vocoder

TL;DR

This work defines general speech restoration (GSR) as the simultaneous removal of multiple distortions and introduces VoiceFixer, a two-stage framework that separates analysis (ResUNet-based mel-spectrogram restoration) from synthesis (TFGAN vocoder) to improve robustness across diverse distortions and sampling rates. By modeling distortions as a composition of several types and training with a comprehensive augmentation pipeline, VoiceFixer demonstrates superior performance over single-task restoration baselines and maintains strong generalization to severely degraded real-world recordings. The approach achieves notable gains in MOS and objective metrics across denoising, dereverberation, declipping, and super-resolution tasks, with the VF-UNet variant delivering the best overall quality, closely approaching an Oracle-Mel bound. The work highlights the value of a two-stage, vocoder-assisted restoration paradigm for broad, real-world speech restoration applications and provides open-source resources to facilitate reproducibility and further research.

Abstract

Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping. However, SSR systems only focus on one task and do not address the general speech restoration problem. In addition, previous SSR systems show limited performance in some speech restoration tasks such as speech super-resolution. To overcome those limitations, we propose a general speech restoration (GSR) task that attempts to remove multiple distortions simultaneously. Furthermore, we propose VoiceFixer, a generative framework to address the GSR task. VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system. We employ a ResUNet to model the analysis stage and a neural vocoder to model the synthesis stage. We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions. Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech enhancement SSR model. VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256. Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches. The source code is available at https://github.com/haoheliu/voicefixer_main.

Figures (12)

• Figure 1: The neural and cognitive model of how human brain understand and restore distorted speech.
• Figure 2: Overview of the proposed VoiceFixer system.
• Figure 3: The architecture of ResUNet, which output has the same size as input.
• Figure 4: The architecture and training scheme of TFGAN, whose generator is later used as vocoder. The generator takes mel spectrogram as input and upsampled it into waveform. Both output waveform and its STFT spectrogram are used to compute loss. We employ both time and frequency discriminators for discriminative training.
• Figure 5: Box plot of the MOS scores on general speech restoration task. Red solid line and green dashed line represent median and mean value.