MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization

TL;DR

MuQ introduces Mel-RVQ as a lightweight, residual-vector-quantized target for self-supervised music representation learning, addressing instability and inefficiency of prior tokenizers. The framework uses a Conformer-based encoder to predict multiple Mel-RVQ tokens per time step, with a simple, offline-trained Mel-RVQ tokenizer and a three-term loss that favors stable code assignments. Scaling from 0.9K to 160K pretraining hours and applying iterative refinement yields state-of-the-art results on MARBLE, while MuQ-MuLan demonstrates strong zero-shot cross-modal music-text tagging. The work provides a practical, scalable path to high-quality music representations and cross-modal embeddings, with open-source code and checkpoints for reproducibility.

Abstract

Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music tagging, instrument classification, key detection, and more. In this paper, we propose a self-supervised music representation learning model for music understanding. Distinguished from previous studies adopting random projection or existing neural codec, the proposed model, named MuQ, is trained to predict tokens generated by Mel Residual Vector Quantization (Mel-RVQ). Our Mel-RVQ utilizes residual linear projection structure for Mel spectrum quantization to enhance the stability and efficiency of target extraction and lead to better performance. Experiments in a large variety of downstream tasks demonstrate that MuQ outperforms previous self-supervised music representation models with only 0.9K hours of open-source pre-training data. Scaling up the data to over 160K hours and adopting iterative training consistently improve the model performance. To further validate the strength of our model, we present MuQ-MuLan, a joint music-text embedding model based on contrastive learning, which achieves state-of-the-art performance in the zero-shot music tagging task on the MagnaTagATune dataset. Code and checkpoints are open source in https://github.com/tencent-ailab/MuQ.

Figures (7)

• Figure 1: Illustration of how the SSL pre-trained model works in music information retrieval (MIR) tasks.
• Figure 2: An overview of our proposed MuQ's framework.
• Figure 3: Comparison between (a) random-projection quantizer, (b)Encodec, and (c) proposed Mel Residual Vector Quantization (Mel-RVQ). Mel-RVQ draws on the lightweight structure of the random-projection quantizer, i.e., it uses only a single linear projection as the encoder, with the Mel spectrum as input. Also, Mel-RVQ borrows the proven effective residual structure from Encodec, but Mel-RVQ further benefits from a much simpler structure than the multiple convolutional layers in Encodec.
• Figure 4: Overall architecture of MuQ-MuLan, a music-text joint embedding model trained with contrastive learning.
• Figure 5: Visualization of radar plot on MARBLE benchmark.