Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Noboru Harada, Nobutaka Ono

Abstract

Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning.

Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

Abstract

Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning.
Paper Structure (17 sections, 1 equation, 2 figures, 3 tables, 1 algorithm)

This paper contains 17 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Masked prediction-based representation learning for audio spectrograms
  4. Masking strategies
  5. Random Masking
  6. Inverse Block Masking
  7. Self-Guided Informed Masking
  8. Proposed Masking Method: DWM
  9. Experiments
  10. Experimental Setup
  11. Comparison of Masking Strategies in Linear Evaluation
  12. Comparison of Masking Strategies in Fine-tuning
  13. Ablations on DWM Hint Ratio Scheduling
  14. Summary of Experiments
  15. Conclusion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Masking examples on an audio spectrogram (hint ratio $r_h=0.0$). Object-centric masking (SGIM) almost entirely covers audio events (objects), whereas the proposed DWM applies dispersion-weighted random masking based on patch-wise spectral variability, yielding masking patterns that loosely align with object-centric behavior.
  • Figure 2: Hint ratio $r_h$ schedules over training epochs for ablation settings.