Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MusicHiFi: Fast High-Fidelity Stereo Vocoding

Ge Zhu, Juan-Pablo Caceres, Zhiyao Duan, Nicholas J. Bryan

TL;DR

The proposed MusicHiFi—an efficient high-fidelity stereophonic vocoder employs a cascade of three generative adversarial networks that convert low-resolution mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth extension, and upmixes to stereophonic audio.

Abstract

Diffusion-based audio and music generation models commonly perform generation by constructing an image representation of audio (e.g., a mel-spectrogram) and then convert it to audio using a phase reconstruction model or vocoder. Typical vocoders, however, produce monophonic audio at lower resolutions (e.g., 16-24 kHz), which limits their usefulness. We propose MusicHiFi -- an efficient high-fidelity stereophonic vocoder. Our method employs a cascade of three generative adversarial networks (GANs) that convert low-resolution mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth extension, and upmixes to stereophonic audio. Compared to past work, we propose 1) a unified GAN-based generator and discriminator architecture and training procedure for each stage of our cascade, 2) a new fast, near downsampling-compatible bandwidth extension module, and 3) a new fast downmix-compatible mono-to-stereo upmixer that ensures the preservation of monophonic content in the output. We evaluate our approach using objective and subjective listening tests and find our approach yields comparable or better audio quality, better spatialization control, and significantly faster inference speed compared to past work. Sound examples are at \url{https://MusicHiFi.github.io/web/}.

MusicHiFi: Fast High-Fidelity Stereo Vocoding

TL;DR

The proposed MusicHiFi—an efficient high-fidelity stereophonic vocoder employs a cascade of three generative adversarial networks that convert low-resolution mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth extension, and upmixes to stereophonic audio.

Abstract

Diffusion-based audio and music generation models commonly perform generation by constructing an image representation of audio (e.g., a mel-spectrogram) and then convert it to audio using a phase reconstruction model or vocoder. Typical vocoders, however, produce monophonic audio at lower resolutions (e.g., 16-24 kHz), which limits their usefulness. We propose MusicHiFi -- an efficient high-fidelity stereophonic vocoder. Our method employs a cascade of three generative adversarial networks (GANs) that convert low-resolution mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth extension, and upmixes to stereophonic audio. Compared to past work, we propose 1) a unified GAN-based generator and discriminator architecture and training procedure for each stage of our cascade, 2) a new fast, near downsampling-compatible bandwidth extension module, and 3) a new fast downmix-compatible mono-to-stereo upmixer that ensures the preservation of monophonic content in the output. We evaluate our approach using objective and subjective listening tests and find our approach yields comparable or better audio quality, better spatialization control, and significantly faster inference speed compared to past work. Sound examples are at \url{https://MusicHiFi.github.io/web/}.
Paper Structure (16 sections, 1 equation, 2 figures, 3 tables)

This paper contains 16 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Overview
  5. MusicHiFi-V
  6. MusicHiFi-BWE
  7. MusicHiFi-M2S
  8. Experiment and Results
  9. Training details
  10. Baselines
  11. Objective evaluation
  12. Objective evaluation result
  13. Subjective evaluation
  14. Subjective evaluation results
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: (a) MusicHiFi inference via a cascade of our vocoder, bandwidth extension (BWE), and mono-to-stereo (M2S) modules that use a shared architecture, but with different weights. (b) MusicHiFi GAN-training for our vocoder, BWE and M2S (top-to-bottom), separately. Our unified inference and training scheme enables novel, high-performing BWE and M2S.
  • Figure 2: Subjective listening test violins plots. BWE test (left) and M2S test (right). Test conditions include (a) AudioSR, (b) cascaded MusicHiFi-V and Aero and (c) MusicHiFi-V+BWE (d) full MusicHiFi (e) MusicHiFi-V+BWE with DSP.