Interpretable and Perceptually-Aligned Music Similarity with Pretrained Embeddings

TL;DR

The paper investigates zero-shot perceptual music similarity by evaluating pretrained text-audio embeddings CLAP and MuQ-MuLan against task-specific similarity models. It introduces an instrument-aware similarity approach that leverages MSS stems to disentangle instrumental contributions, learning per-instrument weights with linear regression. Across the Inst-Sim-ABX benchmark, MuQ-MuLan with a 6-stem Demucs decomposition and ridge regression achieves state-of-the-art perceptual agreement (~90.4%), outperforming both baseline cosine and prior task-specific models while still performing well with ground-truth stems. The work highlights the interpretability and practical potential of instrument-level weighting for perceptual retrieval, and discusses limitations from using synthetic Slakh data and the need for validation on real-world multi-track recordings.

Abstract

Perceptual similarity representations enable music retrieval systems to determine which songs sound most similar to listeners. State-of-the-art approaches based on task-specific training via self-supervised metric learning show promising alignment with human judgment, but are difficult to interpret or generalize due to limited dataset availability. We show that pretrained text-audio embeddings (CLAP and MuQ-MuLan) offer comparable perceptual alignment on similarity tasks without any additional fine-tuning. To surpass this baseline, we introduce a novel method to perceptually align pretrained embeddings with source separation and linear optimization on ABX preference data from listening tests. Our model provides interpretable and controllable instrument-wise weights, allowing music producers to retrieve stem-level loops and samples based on mixed reference songs.