Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios

Xiao-Hang Jiang, Yang Ai, Rui-Chen Zheng, Hui-Peng Du, Ye-Xin Lu, Zhen-Hua Ling

TL;DR

Experimental results confirm that, in scenarios with high sampling rates and low bitrates, the MDCTCodec exhibited high decoded audio quality, improved training and generation efficiency, and compact model size compared to baseline codecs.

Abstract

In this paper, we propose MDCTCodec, an efficient lightweight end-to-end neural audio codec based on the modified discrete cosine transform (MDCT). The encoder takes the MDCT spectrum of audio as input, encoding it into a continuous latent code which is then discretized by a residual vector quantizer (RVQ). Subsequently, the decoder decodes the MDCT spectrum from the quantized latent code and reconstructs audio via inverse MDCT. During the training phase, a novel multi-resolution MDCT-based discriminator (MR-MDCTD) is adopted to discriminate the natural or decoded MDCT spectrum for adversarial training. Experimental results confirm that, in scenarios with high sampling rates and low bitrates, the MDCTCodec exhibited high decoded audio quality, improved training and generation efficiency, and compact model size compared to baseline codecs. Specifically, the MDCTCodec achieved a ViSQOL score of 4.18 at a sampling rate of 48 kHz and a bitrate of 6 kbps on the public VCTK corpus.

MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios

TL;DR

Experimental results confirm that, in scenarios with high sampling rates and low bitrates, the MDCTCodec exhibited high decoded audio quality, improved training and generation efficiency, and compact model size compared to baseline codecs.

Abstract

In this paper, we propose MDCTCodec, an efficient lightweight end-to-end neural audio codec based on the modified discrete cosine transform (MDCT). The encoder takes the MDCT spectrum of audio as input, encoding it into a continuous latent code which is then discretized by a residual vector quantizer (RVQ). Subsequently, the decoder decodes the MDCT spectrum from the quantized latent code and reconstructs audio via inverse MDCT. During the training phase, a novel multi-resolution MDCT-based discriminator (MR-MDCTD) is adopted to discriminate the natural or decoded MDCT spectrum for adversarial training. Experimental results confirm that, in scenarios with high sampling rates and low bitrates, the MDCTCodec exhibited high decoded audio quality, improved training and generation efficiency, and compact model size compared to baseline codecs. Specifically, the MDCTCodec achieved a ViSQOL score of 4.18 at a sampling rate of 48 kHz and a bitrate of 6 kbps on the public VCTK corpus.

Paper Structure

This paper contains 15 sections, 7 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Waveform-Coding-based Neural Audio Codec
  4. Spectral-Coding-based Neural Audio Codec
  5. Proposed Method
  6. Overview
  7. Model Structure
  8. Training Criteria
  9. Experiments
  10. Experimental Setting
  11. Evaluation Metrics
  12. Comparison with Baseline Codecs
  13. Component Analysis
  14. Validation of Generalization
  15. Conclusions

Figures (5)

  • Figure 1: An overview of the proposed MDCTCodec.
  • Figure 2: Details of structures of the encoder and decoder in the MDCTCodec. Here, Conv1D and DeConv1D represent 1D convolutional layer and 1D deconvolutional layer, respectively.
  • Figure 3: Details of the structure of the MR-MDCTD. Here, Conv2D represents 2D convolutional layer.
  • Figure 4: Visualization of natural and MDCTCodec-decoded (at 6 kbps) waveforms and MDCT spectra.
  • Figure 5: Average preference scores (%) of ABX tests on speech quality between MDCTCodec and other codecs (i.e., SoundStream, Encodec, HiFi-Codec, AudioDec, DAC and APCodec), where N/P stands for "no preference" and $p$ denotes the $p$ value of a $t$-test between two codecs.