Self-supervised learning method using multiple sampling strategies for general-purpose audio representation
Ibuki Kuroyanagi, Tatsuya Komatsu
TL;DR
This work tackles the challenge of learning general-purpose audio representations without labels by introducing a self-supervised framework that simultaneously employs three sampling-based contrastive losses: clip-level, frame-level, and task-specific. The final objective $\mathcal{L} = \mathcal{L}_{clip} + \alpha\mathcal{L}_{frame} + \beta\mathcal{L}_{pitch}$ enables a single model to support clip-level tagging, frame-level event detection, and task-specific spectral changes, such as pitch shifts. On a subset of Audioset, the approach outperforms COLA and PANNs across downstream tasks including Google Speech Commands, DCASE SED, and NSynth pitch detection, with bilinear similarity yielding further gains. These results underscore the potential of combining multiple sampling strategies to produce transferable, generalizable audio representations suitable for diverse domains and tasks, and indicate promising directions for scaling to full Audioset and extending to other modalities.
Abstract
We propose a self-supervised learning method using multiple sampling strategies to obtain general-purpose audio representation. Multiple sampling strategies are used in the proposed method to construct contrastive losses from different perspectives and learn representations based on them. In this study, in addition to the widely used clip-level sampling strategy, we introduce two new strategies, a frame-level strategy and a task-specific strategy. The proposed multiple strategies improve the performance of frame-level classification and other tasks like pitch detection, which are not the focus of the conventional single clip-level sampling strategy. We pre-trained the method on a subset of Audioset and applied it to a downstream task with frozen weights. The proposed method improved clip classification, sound event detection, and pitch detection performance by 25%, 20%, and 3.6%.