Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HEAR: Holistic Evaluation of Audio Representations

Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W. Schuller, Christian J. Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, Yonatan Bisk

TL;DR

HEAR introduces a holistic, open benchmark for evaluating general-purpose audio representations across speech, environmental sounds, and music. By forcing a single frozen embedding to support diverse tasks via a simple downstream classifier, it reveals cross-domain strengths and weaknesses of 29 models across 19 tasks. Key findings include a new SOTA on FSD50K by CP-JKU PaSST without fine-tuning and the strong pitch-biased performance of CREPE on pitch-centric tasks, while no model dominates all domains. The work emphasizes reproducibility, modularity, and longitudinal studies, and ends with an open invitation to extend the benchmark to new domains and tasks.

Abstract

What audio embedding approach generalizes best to a wide range of downstream tasks across a variety of everyday domains without fine-tuning? The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. In the spirit of shared exchange, each participant submitted an audio embedding model following a common API that is general-purpose, open-source, and freely available to use. Twenty-nine models by thirteen external teams were evaluated on nineteen diverse downstream tasks derived from sixteen datasets. Open evaluation code, submitted models and datasets are key contributions, enabling comprehensive and reproducible evaluation, as well as previously impossible longitudinal studies. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.

HEAR: Holistic Evaluation of Audio Representations

TL;DR

HEAR introduces a holistic, open benchmark for evaluating general-purpose audio representations across speech, environmental sounds, and music. By forcing a single frozen embedding to support diverse tasks via a simple downstream classifier, it reveals cross-domain strengths and weaknesses of 29 models across 19 tasks. Key findings include a new SOTA on FSD50K by CP-JKU PaSST without fine-tuning and the strong pitch-biased performance of CREPE on pitch-centric tasks, while no model dominates all domains. The work emphasizes reproducibility, modularity, and longitudinal studies, and ends with an open invitation to extend the benchmark to new domains and tasks.

Abstract

What audio embedding approach generalizes best to a wide range of downstream tasks across a variety of everyday domains without fine-tuning? The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios. HEAR evaluates audio representations using a benchmark suite across a variety of domains, including speech, environmental sound, and music. HEAR was launched as a NeurIPS 2021 shared challenge. In the spirit of shared exchange, each participant submitted an audio embedding model following a common API that is general-purpose, open-source, and freely available to use. Twenty-nine models by thirteen external teams were evaluated on nineteen diverse downstream tasks derived from sixteen datasets. Open evaluation code, submitted models and datasets are key contributions, enabling comprehensive and reproducible evaluation, as well as previously impossible longitudinal studies. It still remains an open question whether one single general-purpose audio representation can perform as holistically as the human ear.
Paper Structure (53 sections, 4 figures, 4 tables)

This paper contains 53 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background on representation learning
  3. Inducing representations
  4. Architectures
  5. Training regimes
  6. Using and evaluating representations
  7. HEAR: Holistic Evaluation of Audio Representations
  8. Related audio shared tasks
  9. Evaluation methodology
  10. Evaluation tasks
  11. Models evaluated
  12. Baseline models
  13. wav2vec2
  14. CREPE
  15. OpenL3
  16. ...and 38 more sections

Figures (4)

  • Figure 1: Primary score of submitted models on each HEAR task. Normalized scores are used to show the heat-value of each cell. Missing cells indicate that the model did not successfully complete the task (exhausting GPU memory or exceeding 24 hours downstream training time).
  • Figure 2: t-SNE visualizations of tasks and models, based upon normalized scores. Missing normalized scores were imputed using sklearn's multivariate IterativeImputer.
  • Figure 3: Task versus task correlation scores, based upon normalized scores. Only the highest and lowest correlations are displayed. Cells are sorted to minimize the traveling salesperson distance, mapping correlations [-1, +1] to distances [+2, 0].
  • Figure 4: Model versus model correlation scores, based upon normalized scores. Only the highest and lowest correlations are displayed. Cells are sorted to minimize the traveling salesperson distance, mapping correlations [-1, +1] to distances [+2, 0].