Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sound Check: Auditing Audio Datasets

William Agnew, Julia Barnett, Annie Chu, Rachel Hong, Michael Feffer, Robin Netzorg, Harry H. Jiang, Ezra Awumey, Sauvik Das

TL;DR

It is found that popular generative audio datasets are biased against women, contain toxic stereotypes about marginalized communities, and contain significant amounts of copyrighted work.

Abstract

Generative audio models are rapidly advancing in both capabilities and public utilization -- several powerful generative audio models have readily available open weights, and some tech companies have released high quality generative audio products. Yet, while prior work has enumerated many ethical issues stemming from the data on which generative visual and textual models have been trained, we have little understanding of similar issues with generative audio datasets, including those related to bias, toxicity, and intellectual property. To bridge this gap, we conducted a literature review of hundreds of audio datasets and selected seven of the most prominent to audit in more detail. We found that these datasets are biased against women, contain toxic stereotypes about marginalized communities, and contain significant amounts of copyrighted work. To enable artists to see if they are in popular audio datasets and facilitate exploration of the contents of these datasets, we developed a web tool audio datasets exploration tool at https://audio-audit.vercel.app.

Sound Check: Auditing Audio Datasets

TL;DR

It is found that popular generative audio datasets are biased against women, contain toxic stereotypes about marginalized communities, and contain significant amounts of copyrighted work.

Abstract

Generative audio models are rapidly advancing in both capabilities and public utilization -- several powerful generative audio models have readily available open weights, and some tech companies have released high quality generative audio products. Yet, while prior work has enumerated many ethical issues stemming from the data on which generative visual and textual models have been trained, we have little understanding of similar issues with generative audio datasets, including those related to bias, toxicity, and intellectual property. To bridge this gap, we conducted a literature review of hundreds of audio datasets and selected seven of the most prominent to audit in more detail. We found that these datasets are biased against women, contain toxic stereotypes about marginalized communities, and contain significant amounts of copyrighted work. To enable artists to see if they are in popular audio datasets and facilitate exploration of the contents of these datasets, we developed a web tool audio datasets exploration tool at https://audio-audit.vercel.app.

Paper Structure

This paper contains 39 sections, 12 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Audio AI
  3. Ethics of Audio AI
  4. Dataset Audits
  5. Literature Review of Current Audio Datasets and Models
  6. Literature Review Methodology
  7. Analysis of Current Audio Datasets
  8. Dataset Labeling and Categorization
  9. Distribution and Usage of Datasets
  10. Language Contents of Datasets
  11. Sources of Datasets
  12. Takeaways from Literature Review
  13. Audit of Audio Datasets
  14. Transcription and Textual Analysis
  15. Overview of Audio Datasets
  16. ...and 24 more sections

Figures (12)

  • Figure 1: Flow diagram inspired by PRISMA moher2015preferred detailing the paper corpus used to produce the datasets audited in this paper. We started with 551 records from arXiv (grey) and analyzed 149 full-text articles (blue) to identify 175 unique datasets for analysis (yellow).
  • Figure 2: Stacked bar plot displaying count of times datasets were used by papers in the corpus. Split by Speech, Music, and Non-music/Non-Speech sounds. The vast majority of datasets were only used once, while a select few were used multiple times.
  • Figure 3: Area charts displaying the proportion of all 175 audited datasets by (1) number of times used in our paper corpus, (2) the cumulative total of citations received to date. Split into 4 categories: Speech, Music, Non-Music/Non-Speech sounds, Music and Speech combined.
  • Figure 5: Estimated total duration of audited datasets.
  • Figure 6: Proportion of identity keyword mentions for each dataset. Y-axis is in log-scale.
  • ...and 7 more figures