Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MaskSR: Masked Language Model for Full-band Speech Restoration

Xu Li, Qirui Wang, Xiaoyu Liu

TL;DR

MaskSR introduces a fully generative, end-to-end framework for full-band speech restoration by modeling discrete acoustic tokens with a masked language model conditioned on corrupted input. It uses a frozen neural audio tokenizer (DAC), a speech encoder, and a 2-D codegram LM with iterative MaskGIT-style sampling and classifier-free guidance to jointly address noise, reverberation, clipping, and bandwidth loss. Empirical results on 44.1 kHz restoration and 16 kHz denoising/dereverberation show MaskSR achieving leading bandwidth extension and competitive denoising performance, while preserving speaker identity with guidance. Analyses of codebook modeling indicate parallel codebooks outperform hierarchical approaches, and the STFT-based input representation yields better restoration quality than using DAC for inputs. The work demonstrates the viability of masked LMs for high-fidelity, multimodal speech restoration and lays groundwork for incorporating semantic tokens.

Abstract

Speech restoration aims at restoring high quality speech in the presence of a diverse set of distortions. Although several deep learning paradigms have been studied for this task, the power of the recently emerging language models has not been fully explored. In this paper, we propose MaskSR, a masked language model capable of restoring full-band 44.1 kHz speech jointly considering noise, reverb, clipping, and low bandwidth. MaskSR works with discrete acoustic tokens extracted using a pre-trained neural codec. During training, MaskSR is optimized to predict randomly masked tokens extracted from the high quality target speech, conditioned on the corrupted speech with various distortions. During inference, MaskSR reconstructs the target speech tokens with efficient iterative sampling. Extensive experiments show that MaskSR obtains competitive results on both the full-band speech restoration task and also on sub-tasks compared with a wide range of models.

MaskSR: Masked Language Model for Full-band Speech Restoration

TL;DR

MaskSR introduces a fully generative, end-to-end framework for full-band speech restoration by modeling discrete acoustic tokens with a masked language model conditioned on corrupted input. It uses a frozen neural audio tokenizer (DAC), a speech encoder, and a 2-D codegram LM with iterative MaskGIT-style sampling and classifier-free guidance to jointly address noise, reverberation, clipping, and bandwidth loss. Empirical results on 44.1 kHz restoration and 16 kHz denoising/dereverberation show MaskSR achieving leading bandwidth extension and competitive denoising performance, while preserving speaker identity with guidance. Analyses of codebook modeling indicate parallel codebooks outperform hierarchical approaches, and the STFT-based input representation yields better restoration quality than using DAC for inputs. The work demonstrates the viability of masked LMs for high-fidelity, multimodal speech restoration and lays groundwork for incorporating semantic tokens.

Abstract

Speech restoration aims at restoring high quality speech in the presence of a diverse set of distortions. Although several deep learning paradigms have been studied for this task, the power of the recently emerging language models has not been fully explored. In this paper, we propose MaskSR, a masked language model capable of restoring full-band 44.1 kHz speech jointly considering noise, reverb, clipping, and low bandwidth. MaskSR works with discrete acoustic tokens extracted using a pre-trained neural codec. During training, MaskSR is optimized to predict randomly masked tokens extracted from the high quality target speech, conditioned on the corrupted speech with various distortions. During inference, MaskSR reconstructs the target speech tokens with efficient iterative sampling. Extensive experiments show that MaskSR obtains competitive results on both the full-band speech restoration task and also on sub-tasks compared with a wide range of models.
Paper Structure (16 sections, 1 equation, 3 figures, 5 tables)

This paper contains 16 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Neural Audio Tokenizer
  4. Speech Encoder
  5. Masked Language Model
  6. Other Codebook Modeling Strategies
  7. Experimental Setup
  8. Datasets
  9. Implementation Details
  10. Baseline Models
  11. Evaluation Metrics
  12. Results
  13. Full-band 44.1 kHz Speech Restoration
  14. Wide-band 16 kHz Denoising and Dereverberation
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Training workflow of MaskSR.
  • Figure 2: Effects of guidance on the overall DNSMOS (green) and speaker similarity scores (red).
  • Figure 3: Token classification accuracy of codebook 1 (left) and 3 (right) from MaskSR-S (blue) and SoundStorm (orange). Other codebooks follow the trend of codebook 3.