Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging

Gabriel Meseguer-Brocal, Dorian Desblancs, Romain Hennequin

TL;DR

This paper benchmarks five self-supervised pretext tasks for music tagging using a consistent ResNet-based pipeline and a large unlabeled music catalog. It systematically compares contrastive learning, BYOL, clustering, Barlow Twins, and VICReg, evaluating embeddings with a frozen backbone followed by a simple MLP classifier on five downstream tagging datasets. Across full-data and limited-data regimes, contrastive learning consistently yields the strongest downstream performance, with clustering showing potential but susceptibility to collapse and hyperparameter sensitivity; BYOL, Barlow Twins, and VICReg generally underperform in this context. The study provides practical guidance for selecting pretext tasks in music information retrieval and offers open-source models and code to facilitate further research and deployment.

Abstract

Self-supervised learning has emerged as a powerful way to pre-train generalizable machine learning models on large amounts of unlabeled data. It is particularly compelling in the music domain, where obtaining labeled data is time-consuming, error-prone, and ambiguous. During the self-supervised process, models are trained on pretext tasks, with the primary objective of acquiring robust and informative features that can later be fine-tuned for specific downstream tasks. The choice of the pretext task is critical as it guides the model to shape the feature space with meaningful constraints for information encoding. In the context of music, most works have relied on contrastive learning or masking techniques. In this study, we expand the scope of pretext tasks applied to music by investigating and comparing the performance of new self-supervised methods for music tagging. We open-source a simple ResNet model trained on a diverse catalog of millions of tracks. Our results demonstrate that, although most of these pre-training methods result in similar downstream results, contrastive learning consistently results in better downstream performance compared to other self-supervised pre-training methods. This holds true in a limited-data downstream context.

An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging

TL;DR

This paper benchmarks five self-supervised pretext tasks for music tagging using a consistent ResNet-based pipeline and a large unlabeled music catalog. It systematically compares contrastive learning, BYOL, clustering, Barlow Twins, and VICReg, evaluating embeddings with a frozen backbone followed by a simple MLP classifier on five downstream tagging datasets. Across full-data and limited-data regimes, contrastive learning consistently yields the strongest downstream performance, with clustering showing potential but susceptibility to collapse and hyperparameter sensitivity; BYOL, Barlow Twins, and VICReg generally underperform in this context. The study provides practical guidance for selecting pretext tasks in music information retrieval and offers open-source models and code to facilitate further research and deployment.

Abstract

Self-supervised learning has emerged as a powerful way to pre-train generalizable machine learning models on large amounts of unlabeled data. It is particularly compelling in the music domain, where obtaining labeled data is time-consuming, error-prone, and ambiguous. During the self-supervised process, models are trained on pretext tasks, with the primary objective of acquiring robust and informative features that can later be fine-tuned for specific downstream tasks. The choice of the pretext task is critical as it guides the model to shape the feature space with meaningful constraints for information encoding. In the context of music, most works have relied on contrastive learning or masking techniques. In this study, we expand the scope of pretext tasks applied to music by investigating and comparing the performance of new self-supervised methods for music tagging. We open-source a simple ResNet model trained on a diverse catalog of millions of tracks. Our results demonstrate that, although most of these pre-training methods result in similar downstream results, contrastive learning consistently results in better downstream performance compared to other self-supervised pre-training methods. This holds true in a limited-data downstream context.
Paper Structure (11 sections, 2 figures, 1 table)

This paper contains 11 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Pretext Methodology
  3. Data
  4. Architecture
  5. Pre-training Tasks
  6. Evaluation
  7. Downstream Tasks
  8. Pretext to Downstream Transfer Learning
  9. Limited Data Music Tagging
  10. Training Stability
  11. Conclusion

Figures (2)

  • Figure 1: Downstream results. We apply transfer learning to each task by training an MLP classifier on top of the embeddings generated by the frozen pretext model. We utilize bootstrapping. Each dot represents the metric of a resampled batch. The marker indicates the mean of each result.
  • Figure 2: Limited data results. Each dataset's train set is randomly sampled four times at four different percentages. We report the mean test set metrics obtained for each approach. We also include the results using the full train set in order to showcase each model's performance in comparison to these.