Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training

TL;DR

The paper tackles the data scarcity challenge for transformer-based audio models by introducing MaskSpec, a self-supervised pretraining objective that reconstructs masked spectrogram patches using an asymmetric encoder–decoder. Pretrained on unlabeled AudioSet, MaskSpec achieves strong transfer to diverse downstream tasks, outperforming previous self-supervised and cross-domain approaches in most settings. An ablation study identifies a masking ratio around 75% as particularly effective, and smaller MaskSpec variants also generalize well. Overall, MaskSpec offers a robust, domain-native pretraining strategy for audio transformers and demonstrates practical gains across multiple audio understanding tasks, with code released for reproducibility.

Abstract

Transformer-based models attain excellent results and generalize well when trained on sufficient amounts of data. However, constrained by the limited data available in the audio domain, most transformer-based models for audio tasks are finetuned from pre-trained models in other domains (e.g. image), which has a notable gap with the audio domain. Other methods explore the self-supervised learning approaches directly in the audio domain but currently do not perform well in the downstream tasks. In this paper, we present a novel self-supervised learning method for transformer-based audio models, called masked spectrogram prediction (MaskSpec), to learn powerful audio representations from unlabeled audio data (AudioSet used in this paper). Our method masks random patches of the input spectrogram and reconstructs the masked regions with an encoder-decoder architecture. Without using extra model weights or supervision, experimental results on multiple downstream datasets demonstrate MaskSpec achieves a significant performance gain against the supervised methods and outperforms the previous pre-trained models. In particular, our best model reaches the performance of 0.471 (mAP) on AudioSet, 0.854 (mAP) on OpenMIC2018, 0.982 (accuracy) on ESC-50, 0.976 (accuracy) on SCV2, and 0.823 (accuracy) on DCASE2019 Task1A respectively.

Figures (2)

• Figure 1: Illustration of our proposed MaskSpec method, which is an encoder-decoder framework. The input is the masked spectrogram (only surviving patches reserved), and the output is the reconstructed spectrogram.
• Figure 2: Results of finetuning pre-trained models with different mask ratios on AudioSet.