SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
Xinhao Mei, Gael Le Lan, Haohe Liu, Zhaoheng Ni, Varun Nagaraja, Yang Liu, Yangyang Shi, Vikas Chandra
TL;DR
This work tackles key drawbacks of prior CLAP models by scaling language-audio pretraining to 109 million audio-text pairs, enabling native handling of variable-duration audio up to 30 seconds, and unifying contrastive, self-supervised, and captioning losses in a single-stage pipeline. It introduces a redesigned audio Transformer with alternating local/global attention and sequence packing to efficiently process long, variable-length inputs, paired with a text encoder initialized from ModernBERT. The model is trained with a triad of losses—CLAP, masked patch self-supervision, and caption generation—to learn rich, dense audio representations. Empirical results show SLAP achieves state-of-the-art performance on audio-text retrieval, strong zero-shot classification, and competitive captioning and tagging, underscoring the impact of large-scale, flexible, and multi-objective pretraining for audio understanding.
Abstract
Contrastive language-audio pretraining (CLAP) has achieved notable success in learning semantically rich audio representations and is widely adopted for various audio-related tasks. However, current CLAP models face several key limitations. First, they are typically trained on relatively small datasets, often comprising a few million audio samples. Second, existing CLAP models are restricted to short and fixed duration, which constrains their usage in real-world scenarios with variable-duration audio. Third, the standard contrastive training objective operates on global representations, which may hinder the learning of dense, fine-grained audio features. To address these challenges, we introduce Scalable Language-Audio Pretraining (SLAP), which scales language-audio pretraining to 109 million audio-text pairs with variable audio durations and incorporates multiple training objectives. SLAP unifies contrastive loss with additional self-supervised and captioning losses in a single-stage training, facilitating the learning of richer dense audio representations. The proposed SLAP model achieves new state-of-the-art performance on audio-text retrieval and zero-shot audio classification tasks, demonstrating its effectiveness across diverse benchmarks.