Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenge on Sound Scene Synthesis: Evaluating Text-to-Audio Generation

Junwon Lee, Modan Tailleur, Laurie M. Heller, Keunwoo Choi, Mathieu Lagrange, Brian McFee, Keisuke Imoto, Yuki Okamoto

TL;DR

An evaluation protocol combining objective metric, namely Fr\'echet Audio Distance, with perceptual assessments, utilizing a structured prompt format to enable diverse captions and effective evaluation is presented.

Abstract

Despite significant advancements in neural text-to-audio generation, challenges persist in controllability and evaluation. This paper addresses these issues through the Sound Scene Synthesis challenge held as part of the Detection and Classification of Acoustic Scenes and Events 2024. We present an evaluation protocol combining objective metric, namely Fréchet Audio Distance, with perceptual assessments, utilizing a structured prompt format to enable diverse captions and effective evaluation. Our analysis reveals varying performance across sound categories and model architectures, with larger models generally excelling but innovative lightweight approaches also showing promise. The strong correlation between objective metrics and human ratings validates our evaluation approach. We discuss outcomes in terms of audio quality, controllability, and architectural considerations for text-to-audio synthesizers, providing direction for future research.

Challenge on Sound Scene Synthesis: Evaluating Text-to-Audio Generation

TL;DR

An evaluation protocol combining objective metric, namely Fr\'echet Audio Distance, with perceptual assessments, utilizing a structured prompt format to enable diverse captions and effective evaluation is presented.

Abstract

Despite significant advancements in neural text-to-audio generation, challenges persist in controllability and evaluation. This paper addresses these issues through the Sound Scene Synthesis challenge held as part of the Detection and Classification of Acoustic Scenes and Events 2024. We present an evaluation protocol combining objective metric, namely Fréchet Audio Distance, with perceptual assessments, utilizing a structured prompt format to enable diverse captions and effective evaluation. Our analysis reveals varying performance across sound categories and model architectures, with larger models generally excelling but innovative lightweight approaches also showing promise. The strong correlation between objective metrics and human ratings validates our evaluation approach. We discuss outcomes in terms of audio quality, controllability, and architectural considerations for text-to-audio synthesizers, providing direction for future research.

Paper Structure

This paper contains 11 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Problem and Task Definition
  3. Problem in Current Text-To-Audio Systems
  4. Task Definition
  5. Official Dataset and Baseline System
  6. Evaluation
  7. Results
  8. Conclusion
  9. Challenge Task Overview
  10. Challenge Results
  11. Subjective Evaluation

Figures (10)

  • Figure 1: Performance of various Text-to-Audio models (circled markers: challenge submissions, triangular markers: open-source models) on the evaluation set versus their number of parameters. Color depicts the audio sample rate.
  • Figure 2: Examples that show limited controllability of a recent text-to-audio model (AudioLDM-M audioldm) while controlling sound sources.
  • Figure 3: Correlation between FAD scores on evaluation set and other indicators, computed on the 4 submitted systems and the baseline system.
  • Figure 4: Subjective evaluation results on Foreground Fit. The error bar indicates the standard error.
  • Figure 5: Overview of Sound Scene Synthesis task. A sound synthesis system (i.e., Text-to-Audio model) receives a text prompt as an input, and outputs an audio corresponding to the prompt.
  • ...and 5 more figures