PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

TL;DR

This work introduces PANNs, large-scale pretrained audio neural networks trained on AudioSet, and demonstrates that a Wavegram-Logmel-CNN achieves state-of-the-art 0.439 mAP on AudioSet tagging. It analyzes a broad family of backbones (CNNs, ResNets, MobileNets, 1D CNNs) and novel Wavegram-based representations, along with data balancing and augmentation strategies. The paper further shows that pretrained weights transfer effectively to diverse tasks (ESC-50, DCASE, MSoS, GTZAN, RAVDESS) via fine-tuning or as feature extractors, often surpassing prior methods, especially with limited data. All source code and pretrained models are released, enabling reproducibility and practical adoption of PANNs for wide-ranging audio-pattern recognition tasks.

Abstract

Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification, music classification, speech emotion classification and sound event detection. Recently, neural networks have been applied to tackle audio pattern recognition problems. However, previous systems are built on specific datasets with limited durations. Recently, in computer vision and natural language processing, systems pretrained on large-scale datasets have generalized well to several tasks. However, there is limited research on pretraining systems on large-scale datasets for audio pattern recognition. In this paper, we propose pretrained audio neural networks (PANNs) trained on the large-scale AudioSet dataset. These PANNs are transferred to other audio related tasks. We investigate the performance and computational complexity of PANNs modeled by a variety of convolutional neural networks. We propose an architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and waveform as input feature. Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392. We transfer PANNs to six audio pattern recognition tasks, and demonstrate state-of-the-art performance in several of those tasks. We have released the source code and pretrained models of PANNs: https://github.com/qiuqiangkong/audioset_tagging_cnn.

Figures (12)

• Figure 1: Architecture of Wavegram-Logmel-CNN
• Figure 2: (a) A PANN is pretrained with the AudioSet dataset. (b) For a new task, the PANN is used as a feature extractor. A classifier is built on the extracted embedding features. The shaded rectangle indicates the parameters are frozen and not trained. (c) For a new task, the parameters of a neural network are initialized with a PANN. Then, all parameters are fine-tuned on the new task.
• Figure 3: Class-wise AP of sound events with the CNN14 system. The number inside parentheses indicates the number of training clips. The left, middle, right columns show the AP of sound classes with the number of training clips ranked the 1st to 10th, 250th to 260th and 517th to 527th in the training set of AudioSet.
• Figure 4: Results of PANNs on AudioSet tagging. The transparent and solid lines are training mAP and evaluation mAP, respectively. The six plots show the results with different: (a) architectures; (b) data balancing and data augmentation; (c) embedding size; (d) amount of training data; (e) sampling rate; (f) number of mel bins.
• Figure 5: Multi-adds versus mAP of AudioSet tagging systems. The same types of architectures are grouped in the same color.