Masked Audio Modeling with CLAP and Multi-Objective Learning
Yifei Xin, Xiulian Peng, Yan Lu
TL;DR
This paper tackles the limited semantic modeling of masked audio modeling (MAM) by introducing MAM-CLAP, which distills cross-modal knowledge from CLAP into both visible and masked spectrogram patches, and SupMAM, a multi-objective extension that adds a supervised classification branch to learn global spectrogram features. The authors formulate a loss that combines CLAP-guided reconstruction with a classification loss, balancing them with a weight λ_cls, and show that using CLAP targets on both patch types improves semantic representations. Through extensive experiments on AudioSet, ESC-50, and Speech Commands, MAM-CLAP achieves state-of-the-art performance, and when combined with SupMAM (SupMAM-CLAP) sets new records across multiple audio and speech classification benchmarks. The results demonstrate the practical value of cross-modal supervision and multi-objective learning for robust, semantically meaningful audio representations.
Abstract
Most existing masked audio modeling (MAM) methods learn audio representations by masking and reconstructing local spectrogram patches. However, the reconstruction loss mainly accounts for the signal-level quality of the reconstructed spectrogram and is still limited in extracting high-level audio semantics. In this paper, we propose to enhance the semantic modeling of MAM by distilling cross-modality knowledge from contrastive language-audio pretraining (CLAP) representations for both masked and unmasked regions (MAM-CLAP) and leveraging a multi-objective learning strategy with a supervised classification branch (SupMAM), thereby providing more semantic knowledge for MAM and enabling it to effectively learn global features from labels. Experiments show that our methods significantly improve the performance on multiple downstream tasks. Furthermore, by combining our MAM-CLAP with SupMAM, we can achieve new state-of-the-art results on various audio and speech classification tasks, exceeding previous self-supervised learning and supervised pretraining methods.