Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Deep Learning Audio Generation Methods

Matej Božić, Marko Horvat

TL;DR

This survey compiles the state of deep learning for audio generation, detailing three core ingredients: representations (raw waveform, mel-spectrogram, neural codecs), architectures (auto-encoders, GANs, normalizing flows, transformers, diffusion models), and evaluation metrics (human MOS, IS/FID/FAD, KL divergence). It emphasizes the rise of transformer and diffusion frameworks as dominant approaches, discusses their trade-offs (e.g., sampling speed, context length, and conditioning), and surveys representative models and datasets across TTS, music, and broader audio generation tasks. The work highlights practical implications for researchers, including the move from hand-crafted features to learned representations, token-based neural codecs, and end-to-end systems, underpinned by evaluation protocols that balance perceptual quality with statistical fidelity. Overall, the paper maps a field transitioning toward multi-task and multi-modality models powered by large-scale data and compute, guiding future directions in scalable, high-fidelity audio generation.

Abstract

This article presents a review of typical techniques used in three distinct aspects of deep learning model development for audio generation. In the first part of the article, we provide an explanation of audio representations, beginning with the fundamental audio waveform. We then progress to the frequency domain, with an emphasis on the attributes of human hearing, and finally introduce a relatively recent development. The main part of the article focuses on explaining basic and extended deep learning architecture variants, along with their practical applications in the field of audio generation. The following architectures are addressed: 1) Autoencoders 2) Generative adversarial networks 3) Normalizing flows 4) Transformer networks 5) Diffusion models. Lastly, we will examine four distinct evaluation metrics that are commonly employed in audio generation. This article aims to offer novice readers and beginners in the field a comprehensive understanding of the current state of the art in audio generation methods as well as relevant studies that can be explored for future research.

A Survey of Deep Learning Audio Generation Methods

TL;DR

This survey compiles the state of deep learning for audio generation, detailing three core ingredients: representations (raw waveform, mel-spectrogram, neural codecs), architectures (auto-encoders, GANs, normalizing flows, transformers, diffusion models), and evaluation metrics (human MOS, IS/FID/FAD, KL divergence). It emphasizes the rise of transformer and diffusion frameworks as dominant approaches, discusses their trade-offs (e.g., sampling speed, context length, and conditioning), and surveys representative models and datasets across TTS, music, and broader audio generation tasks. The work highlights practical implications for researchers, including the move from hand-crafted features to learned representations, token-based neural codecs, and end-to-end systems, underpinned by evaluation protocols that balance perceptual quality with statistical fidelity. Overall, the paper maps a field transitioning toward multi-task and multi-modality models powered by large-scale data and compute, guiding future directions in scalable, high-fidelity audio generation.

Abstract

This article presents a review of typical techniques used in three distinct aspects of deep learning model development for audio generation. In the first part of the article, we provide an explanation of audio representations, beginning with the fundamental audio waveform. We then progress to the frequency domain, with an emphasis on the attributes of human hearing, and finally introduce a relatively recent development. The main part of the article focuses on explaining basic and extended deep learning architecture variants, along with their practical applications in the field of audio generation. The following architectures are addressed: 1) Autoencoders 2) Generative adversarial networks 3) Normalizing flows 4) Transformer networks 5) Diffusion models. Lastly, we will examine four distinct evaluation metrics that are commonly employed in audio generation. This article aims to offer novice readers and beginners in the field a comprehensive understanding of the current state of the art in audio generation methods as well as relevant studies that can be explored for future research.
Paper Structure (21 sections, 2 figures)

This paper contains 21 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. Text-to-Speech
  5. Music generation
  6. Audio features
  7. Raw waveform
  8. Mel-spectrogram
  9. Neural codecs
  10. Architectures
  11. Auto-encoders
  12. Generative adversarial networks
  13. Normalizing flows
  14. Transformer networks
  15. Diffusion models
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The main sections of the survey.
  • Figure 2: Two deep learning architectures that appear to have little in common until we look closer. The generator mimics the auto-encoder's decoder, whereas the discriminator resembles the encoder.