Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

Junhyeok Lee, Seungu Han

TL;DR

NU-Wave presents a conditional diffusion probabilistic model for neural audio upsampling that achieves 48 kHz waveform synthesis from 16/24 kHz inputs. By integrating a diffusion-based vocoder with a downsampled conditioning signal and a continuous noise-level training regime, it delivers superior SNR and LSD with far fewer parameters than prior approaches. The method demonstrates strong performance on the VCTK dataset for both single- and multi-speaker scenarios and shows outputs that are nearly indistinguishable from references in ABX tests, while remaining computationally efficient. The work establishes diffusion models as a viable and effective framework for high-fidelity audio bandwidth extension and upsampling.

Abstract

In this work, we introduce NU-Wave, the first neural audio upsampling model to produce waveforms of sampling rate 48kHz from coarse 16kHz or 24kHz inputs, while prior works could generate only up to 16kHz. NU-Wave is the first diffusion probabilistic model for audio super-resolution which is engineered based on neural vocoders. NU-Wave generates high-quality audio that achieves high performance in terms of signal-to-noise ratio (SNR), log-spectral distance (LSD), and accuracy of the ABX test. In all cases, NU-Wave outperforms the baseline models despite the substantially smaller model capacity (3.0M parameters) than baselines (5.4-21%). The audio samples of our model are available at https://mindslab-ai.github.io/nuwave, and the code will be made available soon.

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

TL;DR

NU-Wave presents a conditional diffusion probabilistic model for neural audio upsampling that achieves 48 kHz waveform synthesis from 16/24 kHz inputs. By integrating a diffusion-based vocoder with a downsampled conditioning signal and a continuous noise-level training regime, it delivers superior SNR and LSD with far fewer parameters than prior approaches. The method demonstrates strong performance on the VCTK dataset for both single- and multi-speaker scenarios and shows outputs that are nearly indistinguishable from references in ABX tests, while remaining computationally efficient. The work establishes diffusion models as a viable and effective framework for high-fidelity audio bandwidth extension and upsampling.

Abstract

In this work, we introduce NU-Wave, the first neural audio upsampling model to produce waveforms of sampling rate 48kHz from coarse 16kHz or 24kHz inputs, while prior works could generate only up to 16kHz. NU-Wave is the first diffusion probabilistic model for audio super-resolution which is engineered based on neural vocoders. NU-Wave generates high-quality audio that achieves high performance in terms of signal-to-noise ratio (SNR), log-spectral distance (LSD), and accuracy of the ABX test. In all cases, NU-Wave outperforms the baseline models despite the substantially smaller model capacity (3.0M parameters) than baselines (5.4-21%). The audio samples of our model are available at https://mindslab-ai.github.io/nuwave, and the code will be made available soon.

Paper Structure

This paper contains 17 sections, 7 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Diffusion probabilistic models
  4. Conditional diffusion models as a neural vocoder
  5. Approach
  6. Network architecture
  7. Training objective
  8. Noise schedule
  9. Experiments
  10. Dataset
  11. Training
  12. Evaluation
  13. Baselines
  14. Implementation details
  15. Results
  16. ...and 2 more sections

Figures (3)

  • Figure 1: The network architecture of NU-Wave. Noisy speech $y_{\bar{\alpha}}$, downsampled speech $y_{d}$, and noise level $\sqrt{\bar{\alpha}}$ are inputs of model. The model estimates noise $\hat{\epsilon}$ to reconstruct $y_0$ from $y_{\bar{\alpha}}$.
  • Figure 2: Spectrograms of reference and upsampled speeches. Red lines indicate the Nyquist frequency of downsampled signals. (a1-a5) are samples of $r=2$, MultiSpeaker (p360_001) and (b1-b5) are samples of $r=3$, SingleSpeaker (p225_359). More samples are available at https://mindslab-ai.github.io/nuwave
  • Figure : Training.