Linear Complexity Self-Supervised Learning for Music Understanding with Random Quantizer

TL;DR

The paper tackles the high computational burden of foundation models in music information retrieval by proposing a compact architecture built on Branchformer with SummaryMixing and a non-trainable random tokenizer. Through pre-training on public and a large proprietary music corpus, the approach achieves competitive MIR performance while reducing parameters by $8.5\%$ to $12.3\%$, and even surpasses some baselines on tasks like instrument classification and singer identification. The study demonstrates that linear-complexity attention and random quantization can preserve representational power in music tasks, bringing practical benefits for scalability. Overall, the work strengthens the case for scalable, self-supervised foundation models in MIR and highlights avenues for further efficiency gains and cross-domain integration.

Abstract

In recent years, foundation models have become very popular due to their exceptional performance, mainly in natural language (NLP) tasks where they were first introduced. These models usually consist of hundreds of millions, or even billions, of parameters, making them resource-intensive during training and in production systems, leading to increased costs. This paper focuses on the reduction of a foundation's model size when applied to music information retrieval (MIR) tasks. Our research combines the Branchformer architecture with SummaryMixing, which were first applied in speech recognition, along with a random quantization process. To facilitate reproducibility, we conduct pre-training on publicly available datasets, complemented by a proprietary dataset comparable in scale to other private datasets reported in the literature. We ensure robust evaluation by using a framework consisting of a variety of downstream MIR tasks. Our results show that our architecture achieves competitive performance when compared with other state-of-the-art models that use multi-head self-attention, while reducing the model size from 8.5% up to 12.3%.

Figures (1)

• Figure 1: An illustration of our framework is shown here. On the left, we present the Branchformer architecture, which takes raw masked audio as input and generates two distinct sets of logits. Specifically, $m_0, ..., m_{k}$ represent the logits corresponding to the mel spectrogram, while $z_0, ..., z_{k}$ correspond to the logits for the quantized tokens. On the right, we show the random tokenizer, which receives the mel spectrogram as input and produces the quantized tokens. Importantly, only the masked tokens are utilized for the loss calculation, ensuring that the model focuses on the relevant parts of the input.