Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
Sarthak Yadav, Zheng-Hua Tan
TL;DR
Audio Mamba (SSAM) introduces a selective state-space approach for self-supervised audio representation learning from masked spectrogram patches. By coupling context-aware Mamba blocks with a reconstruction objective trained on AudioSet, SSAM achieves substantial improvements over self-supervised SSAST baselines across ten downstream tasks, demonstrating strong data efficiency and robustness to patch size and sequence length. The method produces latent representations suitable for downstream classifiers while maintaining a lightweight reconstruction head during pretraining. Overall, SSAM shows that selective state-space models are a viable pathway for general-purpose, long-range audio representation learning with practical benefits for real-world tasks.
Abstract
Despite its widespread adoption as the prominent neural architecture, the Transformer has spurred several independent lines of work to address its limitations. One such approach is selective state space models, which have demonstrated promising results for language modelling. However, their feasibility for learning self-supervised, general-purpose audio representations is yet to be investigated. This work proposes Audio Mamba, a selective state space model for learning general-purpose audio representations from randomly masked spectrogram patches through self-supervision. Empirical results on ten diverse audio recognition downstream tasks show that the proposed models, pretrained on the AudioSet dataset, consistently outperform comparable self-supervised audio spectrogram transformer (SSAST) baselines by a considerable margin and demonstrate better performance in dataset size, sequence length and model size comparisons.