Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wave-U-Mamba: An End-To-End Framework For High-Quality And Efficient Speech Super Resolution

Yongjoon Lee, Chanwoo Kim

TL;DR

The paper tackles Speech Super-Resolution (SSR) by reconstructing high-resolution waveform directly from low-sampling-rate speech, avoiding mel-based representations and vocoders. It introduces Wave-U-Mamba, an end-to-end GAN-based waveform model with a U-Net backbone and Mamba-based down/up blocks, trained with Mel loss, multi-resolution STFT loss, and GAN losses. On the VCTK-test, Wave-U-Mamba achieves state-of-the-art LSD across LR inputs from $8$ to $24$ kHz to a target $48$ kHz and attains superior MOS compared to strong baselines. The approach uses fewer than $2 ext{\%}$ of the parameters and delivers up to ~536× faster inference on an A100 GPU, enabling highly efficient high-quality SSR and reducing reliance on vocoders and phase reconstruction.

Abstract

Speech Super-Resolution (SSR) is a task of enhancing low-resolution speech signals by restoring missing high-frequency components. Conventional approaches typically reconstruct log-mel features, followed by a vocoder that generates high-resolution speech in the waveform domain. However, as mel features lack phase information, this can result in performance degradation during the reconstruction phase. Motivated by recent advances with Selective State Spaces Models (SSMs), we propose a method, referred to as Wave-U-Mamba that directly performs SSR in time domain. In our comparative study, including models such as WSRGlow, NU-Wave 2, and AudioSR, Wave-U-Mamba demonstrates superior performance, achieving the lowest Log-Spectral Distance (LSD) across various low-resolution sampling rates, ranging from 8 to 24 kHz. Additionally, subjective human evaluations, scored using Mean Opinion Score (MOS) reveal that our method produces SSR with natural and human-like quality. Furthermore, Wave-U-Mamba achieves these results while generating high-resolution speech over nine times faster than baseline models on a single A100 GPU, with parameter sizes less than 2\% of those in the baseline models.

Wave-U-Mamba: An End-To-End Framework For High-Quality And Efficient Speech Super Resolution

TL;DR

The paper tackles Speech Super-Resolution (SSR) by reconstructing high-resolution waveform directly from low-sampling-rate speech, avoiding mel-based representations and vocoders. It introduces Wave-U-Mamba, an end-to-end GAN-based waveform model with a U-Net backbone and Mamba-based down/up blocks, trained with Mel loss, multi-resolution STFT loss, and GAN losses. On the VCTK-test, Wave-U-Mamba achieves state-of-the-art LSD across LR inputs from to kHz to a target kHz and attains superior MOS compared to strong baselines. The approach uses fewer than of the parameters and delivers up to ~536× faster inference on an A100 GPU, enabling highly efficient high-quality SSR and reducing reliance on vocoders and phase reconstruction.

Abstract

Speech Super-Resolution (SSR) is a task of enhancing low-resolution speech signals by restoring missing high-frequency components. Conventional approaches typically reconstruct log-mel features, followed by a vocoder that generates high-resolution speech in the waveform domain. However, as mel features lack phase information, this can result in performance degradation during the reconstruction phase. Motivated by recent advances with Selective State Spaces Models (SSMs), we propose a method, referred to as Wave-U-Mamba that directly performs SSR in time domain. In our comparative study, including models such as WSRGlow, NU-Wave 2, and AudioSR, Wave-U-Mamba demonstrates superior performance, achieving the lowest Log-Spectral Distance (LSD) across various low-resolution sampling rates, ranging from 8 to 24 kHz. Additionally, subjective human evaluations, scored using Mean Opinion Score (MOS) reveal that our method produces SSR with natural and human-like quality. Furthermore, Wave-U-Mamba achieves these results while generating high-resolution speech over nine times faster than baseline models on a single A100 GPU, with parameter sizes less than 2\% of those in the baseline models.
Paper Structure (15 sections, 5 equations, 3 figures, 3 tables)

This paper contains 15 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Selective State Spaces Models
  4. Time Domain Speech Processing
  5. Wave-U-Mamba
  6. Task Definition
  7. Generator
  8. Discriminator
  9. Training Objectives
  10. Experimental Results
  11. Training
  12. Evaluation
  13. Results
  14. Conclusion
  15. Acknowledgements

Figures (3)

  • Figure 1: The architecture of DownsampleBlock (Left), Wave-U-Mamba Generator (Middle), and UpsampleBlock (Right).
  • Figure 2: The architecture of MambaBlock (Left) and ResidualBlock (Right).
  • Figure 3: Enhanced mel spectrograms using Wave-U-Mamba at different training phases. \ref{['subfig:a']} and \ref{['subfig:b']} represent Low-Resolution and High-Resolution mel spectrogram of the ground truth speech signal. \ref{['subfig:c']} and \ref{['subfig:d']} represent recovered mel spectrogram from epoch 2 and 9, respectively.