Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VGGSound: A Large-scale Audio-Visual Dataset

Honglie Chen, Weidi Xie, Andrea Vedaldi, Andrew Zisserman

TL;DR

The paper introduces a scalable, CV-driven pipeline to assemble a large-scale audio-visual dataset from unconstrained YouTube content with low label noise. The resulting VGG-Sound dataset contains over 200k video clips across 309 classes and includes audio-visual correspondences verified through visual and audio checks, enabling robust audio recognition benchmarks. The authors establish baselines using CNN-based architectures on spectrogram inputs, demonstrating that deeper models and advanced pooling like NetVLAD improve performance, and they provide analyses on both in-domain and AV-grounded test sets. This work offers a practical, scalable resource for advancing audio recognition and multi-modal audio-visual understanding in realistic, noisy environments.

Abstract

Our goal is to collect a large-scale audio-visual dataset with low label noise from videos in the wild using computer vision techniques. The resulting dataset can be used for training and evaluating audio recognition models. We make three contributions. First, we propose a scalable pipeline based on computer vision techniques to create an audio dataset from open-source media. Our pipeline involves obtaining videos from YouTube; using image classification algorithms to localize audio-visual correspondence; and filtering out ambient noise using audio verification. Second, we use this pipeline to curate the VGGSound dataset consisting of more than 210k videos for 310 audio classes. Third, we investigate various Convolutional Neural Network~(CNN) architectures and aggregation approaches to establish audio recognition baselines for our new dataset. Compared to existing audio datasets, VGGSound ensures audio-visual correspondence and is collected under unconstrained conditions. Code and the dataset are available at http://www.robots.ox.ac.uk/~vgg/data/vggsound/

VGGSound: A Large-scale Audio-Visual Dataset

TL;DR

The paper introduces a scalable, CV-driven pipeline to assemble a large-scale audio-visual dataset from unconstrained YouTube content with low label noise. The resulting VGG-Sound dataset contains over 200k video clips across 309 classes and includes audio-visual correspondences verified through visual and audio checks, enabling robust audio recognition benchmarks. The authors establish baselines using CNN-based architectures on spectrogram inputs, demonstrating that deeper models and advanced pooling like NetVLAD improve performance, and they provide analyses on both in-domain and AV-grounded test sets. This work offers a practical, scalable resource for advancing audio recognition and multi-modal audio-visual understanding in realistic, noisy environments.

Abstract

Our goal is to collect a large-scale audio-visual dataset with low label noise from videos in the wild using computer vision techniques. The resulting dataset can be used for training and evaluating audio recognition models. We make three contributions. First, we propose a scalable pipeline based on computer vision techniques to create an audio dataset from open-source media. Our pipeline involves obtaining videos from YouTube; using image classification algorithms to localize audio-visual correspondence; and filtering out ambient noise using audio verification. Second, we use this pipeline to curate the VGGSound dataset consisting of more than 210k videos for 310 audio classes. Third, we investigate various Convolutional Neural Network~(CNN) architectures and aggregation approaches to establish audio recognition baselines for our new dataset. Compared to existing audio datasets, VGGSound ensures audio-visual correspondence and is collected under unconstrained conditions. Code and the dataset are available at http://www.robots.ox.ac.uk/~vgg/data/vggsound/

Paper Structure

This paper contains 9 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The VGG-Sound dataset
  4. Experiments
  5. Experimental Setup and Evaluation
  6. Implementation Details
  7. Results
  8. Conclusion
  9. List of VGG-Sound Classes

Figures (1)

  • Figure 1: The top row in this figure shows example video frame and audio pairs of VGG-Sound classes, the bottom bar chart demostrates VGG-Sound classes with sizes of each audio class sorted by descending order.