Masked Modeling Duo: Towards a Universal Audio Pre-training Framework

TL;DR

This paper introduces Masked Modeling Duo (M2D), a self-supervised audio pre-training method that encodes masked and visible input portions separately to generate a training signal from the masked part, improving masked-prediction SSL. It further extends the framework to M2D-X, which adds an offline network, background noise, and configurable tasks to learn specialized representations for application domains, including scenarios with small data. Empirical results show that M2D delivers strong general-purpose audio representations that rival or surpass baselines on AudioSet and speech tasks, while M2D-X achieves top-tier performance in specialized domains such as AudioSet, speech, and medical audio with limited data. The work demonstrates the potential of a universal audio pre-training framework, with practical implications and publicly available code and pretrained weights to facilitate future research and deployment.

Abstract

Self-supervised learning (SSL) using masked prediction has made great strides in general-purpose audio representation. This study proposes Masked Modeling Duo (M2D), an improved masked prediction SSL, which learns by predicting representations of masked input signals that serve as training signals. Unlike conventional methods, M2D obtains a training signal by encoding only the masked part, encouraging the two networks in M2D to model the input. While M2D improves general-purpose audio representations, a specialized representation is essential for real-world applications, such as in industrial and medical domains. The often confidential and proprietary data in such domains is typically limited in size and has a different distribution from that in pre-training datasets. Therefore, we propose M2D for X (M2D-X), which extends M2D to enable the pre-training of specialized representations for an application X. M2D-X learns from M2D and an additional task and inputs background noise. We make the additional task configurable to serve diverse applications, while the background noise helps learn on small data and forms a denoising task that makes representation robust. With these design choices, M2D-X should learn a representation specialized to serve various application needs. Our experiments confirmed that the representations for general-purpose audio, specialized for the highly competitive AudioSet and speech domain, and a small-data medical task achieve top-level performance, demonstrating the potential of using our models as a universal audio pre-training framework. Our code is available online for future studies at https://github.com/nttcslab/m2d

Figures (6)

• Figure 1: (a) M2D overview. The online network encodes the visible patches of the randomly masked input signal, while the target network encodes the masked patches. M2D encourages both representations to model the input signal through a task to predict the target representation using the online representation. (b) M2D-X overview. Introducing background noise and offline network allows M2D-X to learn from M2D, denoising, and an additional task, enabling pre-training of specialized representations for diverse applications.
• Figure 2: M2D pre-training flow. The online network encodes the visible patch $x_v$ and predicts the representation $\tilde{z}_m$ of the masked patch $x_m$ encoded in the target network. M2D learns the representation by maximizing the agreement between the prediction $\hat{z}_m$ and $\tilde{z}_m$.
• Figure 3: M2D-X pre-training flow and application scenarios. M2D-X adds an offline network to M2D and inputs background noise, forming a multitask of M2D, a denoising task, and an additional task. Scenario examples show the offline network configurations to compose an additional task for each purpose: supervised learning of task labels, distilling a domain pre-trained model, or regularizing to prevent overfitting M2D to a small dataset.
• Figure 4: Evaluation \ref{['sec:eval-general']} General-purpose audio representation: Masking ratio ablation average linear evaluation results (%).
• Figure 5: Evaluation \ref{['sec:eval-icbhi']} Specialized Representation for ICBHI2017: Comparing the progress of Score (%) among further pre-training settings.