Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation

Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino

TL;DR

Masked Spectrogram Modeling (MSM) uses Masked Autoencoders to learn general-purpose audio representations by reconstructing masked spectrogram patches from the input itself. The approach partitions spectrograms into time-frequency patches and trains on a large AudioSet corpus, evaluating on 15 HEAR 2021 tasks across speech, environmental sound, and music domains. MSM-MAE achieves competitive performance, outperforming several baselines on seven tasks and offering design-choice insights (e.g., longer durations, finer time resolution) validated by qualitative visualizations of reconstructions and attention maps. The work demonstrates the viability of input-driven, patch-based pretraining for scalable, domain-general audio representations and provides code for reproducibility.

Abstract

Recent general-purpose audio representations show state-of-the-art performance on various audio tasks. These representations are pre-trained by self-supervised learning methods that create training signals from the input. For example, typical audio contrastive learning uses temporal relationships among input sounds to create training signals, whereas some methods use a difference among input views created by data augmentations. However, these training signals do not provide information derived from the intact input sound, which we think is suboptimal for learning representation that describes the input as it is. In this paper, we seek to learn audio representations from the input itself as supervision using a pretext task of auto-encoding of masked spectrogram patches, Masked Spectrogram Modeling (MSM, a variant of Masked Image Modeling applied to audio spectrogram). To implement MSM, we use Masked Autoencoders (MAE), an image self-supervised learning method. MAE learns to efficiently encode the small number of visible patches into latent representations to carry essential information for reconstructing a large number of masked patches. While training, MAE minimizes the reconstruction error, which uses the input as training signal, consequently achieving our goal. We conducted experiments on our MSM using MAE (MSM-MAE) models under the evaluation benchmark of the HEAR 2021 NeurIPS Challenge. Our MSM-MAE models outperformed the HEAR 2021 Challenge results on seven out of 15 tasks (e.g., accuracies of 73.4% on CREMA-D and 85.8% on LibriCount), while showing top performance on other tasks where specialized models perform better. We also investigate how the design choices of MSM-MAE impact the performance and conduct qualitative analysis of visualization outcomes to gain an understanding of learned representations. We make our code available online.

Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation

TL;DR

Masked Spectrogram Modeling (MSM) uses Masked Autoencoders to learn general-purpose audio representations by reconstructing masked spectrogram patches from the input itself. The approach partitions spectrograms into time-frequency patches and trains on a large AudioSet corpus, evaluating on 15 HEAR 2021 tasks across speech, environmental sound, and music domains. MSM-MAE achieves competitive performance, outperforming several baselines on seven tasks and offering design-choice insights (e.g., longer durations, finer time resolution) validated by qualitative visualizations of reconstructions and attention maps. The work demonstrates the viability of input-driven, patch-based pretraining for scalable, domain-general audio representations and provides code for reproducibility.

Abstract

Recent general-purpose audio representations show state-of-the-art performance on various audio tasks. These representations are pre-trained by self-supervised learning methods that create training signals from the input. For example, typical audio contrastive learning uses temporal relationships among input sounds to create training signals, whereas some methods use a difference among input views created by data augmentations. However, these training signals do not provide information derived from the intact input sound, which we think is suboptimal for learning representation that describes the input as it is. In this paper, we seek to learn audio representations from the input itself as supervision using a pretext task of auto-encoding of masked spectrogram patches, Masked Spectrogram Modeling (MSM, a variant of Masked Image Modeling applied to audio spectrogram). To implement MSM, we use Masked Autoencoders (MAE), an image self-supervised learning method. MAE learns to efficiently encode the small number of visible patches into latent representations to carry essential information for reconstructing a large number of masked patches. While training, MAE minimizes the reconstruction error, which uses the input as training signal, consequently achieving our goal. We conducted experiments on our MSM using MAE (MSM-MAE) models under the evaluation benchmark of the HEAR 2021 NeurIPS Challenge. Our MSM-MAE models outperformed the HEAR 2021 Challenge results on seven out of 15 tasks (e.g., accuracies of 73.4% on CREMA-D and 85.8% on LibriCount), while showing top performance on other tasks where specialized models perform better. We also investigate how the design choices of MSM-MAE impact the performance and conduct qualitative analysis of visualization outcomes to gain an understanding of learned representations. We make our code available online.
Paper Structure (24 sections, 1 equation, 8 figures, 7 tables)

This paper contains 24 sections, 1 equation, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Masked Autoencoders (MAE)
  4. Masked Spectrogram Modeling using Masked Autoencoders
  5. Input Audio Duration
  6. Patch Size
  7. Feature Calculation for Downstream Tasks
  8. Experiments
  9. Experimental Details
  10. Pre-training
  11. Evaluation
  12. Downstream Tasks
  13. Experimental Results: Comparison with the HEAR 2021 Results
  14. Experimental Results: Impact of Design Choices on Performance
  15. Impact of Input Audio Duration
  16. ...and 9 more sections

Figures (8)

  • Figure 1: MAE he2021masked pre-training flow; we redraw Figure 1 in the MAE paper, in which we replaced the input image with a spectrogram and added loss calculation flow. MAE masks 75% of input patches, then the encoder processes the visible 25% of patches only, saving 75% of computation load. The lightweight decoder takes as input the encoded 25% plus mask tokens that fill the masked 75% of the input, then reconstructs spectrogram. The loss calculates the reconstruction error of the masked patches, which is a mean squared error (MSE).
  • Figure 2: Reconstructions for the pre-trained MSM-MAE-304 with a mask ratio of 0.75, obtained from three attempts of reconstruction of three sounds. Each example shows the input, the reconstruction result, and the difference between them, the error (RMS); the darker the color, the higher the reconstruction error. The white squares in the reconstruction results show the visible patch.
  • Figure 3: Reconstructions with three patterns of masks, showing the difference in the error under different availability of visible information along axes. The mask ratio is 0.5, i.e., half of the patches are masked for all examples.
  • Figure 4: Reconstructions with various mask ratios, focusing on cases of extremely small numbers of visible patches. We show white squares, the visible patches, on the results with mask ratios higher than 0.75 only to improve visibility. MSE is the mean squared error of reconstruction averaged over the entire spectrogram.
  • Figure 5: MSM-MAE encoder self-attention map for reference points (2, 7) and (1, 12).
  • ...and 3 more figures