Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings

TL;DR

The paper tackles scalable music cover retrieval by leveraging lyrics as a stable invariant across renditions. It introduces LIVI, which first constructs a lyrics-informed embedding space via an ASR-based transcription followed by a multilingual text encoder, then trains a lightweight audio encoder to project audio into that fixed space, eliminating transcription at inference. The approach achieves retrieval performance on par with or surpassing state-of-the-art harmonic-based systems while delivering large efficiency gains, notably reducing end-to-end latency by 3.2× and inference time by ~20× compared with transcription-based methods. Experiments on large-scale benchmarks ( Covers80, SHS100k, Discogs-VI ) demonstrate strong accuracy, robust lyric alignment, and practical scalability, with limitations including reliance on vocal content and a proprietary vocal-detection preprocessing step. The method offers a reproducible, domain-grounded alternative to complex multimodal pipelines and suggests potential gains from combining lyrics with harmonic cues in future work.

Abstract

Music Cover Retrieval, also known as Version Identification, aims to recognize distinct renditions of the same underlying musical work, a task central to catalog management, copyright enforcement, and music retrieval. State-of-the-art approaches have largely focused on harmonic and melodic features, employing increasingly complex audio pipelines designed to be invariant to musical attributes that often vary widely across covers. While effective, these methods demand substantial training time and computational resources. By contrast, lyrics constitute a strong invariant across covers, though their use has been limited by the difficulty of extracting them accurately and efficiently from polyphonic audio. Early methods relied on simple frameworks that limited downstream performance, while more recent systems deliver stronger results but require large models integrated within complex multimodal architectures. We introduce LIVI (Lyrics-Informed Version Identification), an approach that seeks to balance retrieval accuracy with computational efficiency. First, LIVI leverages supervision from state-of-the-art transcription and text embedding models during training to achieve retrieval accuracy on par with--or superior to--harmonic-based systems. Second, LIVI remains lightweight and efficient by removing the transcription step at inference, challenging the dominance of complexity-heavy pipelines.

Figures (3)

• Figure 1: Overview of the proposed LIVI framework. (a) A frozen text encoder ($g_{\text{text}}$) combines an ASR model with a pre-trained text embedding model to produce lyrics embeddings $t_i$. (b) An audio encoder ($g_{\text{audio}}$) projects ASR encoder latent representations into the same embedding space. (c) Training optimizes a combined objective: pointwise alignment of $a_i$ with $t_i$ under cosine similarity, and geometry preservation ensuring that pairwise similarities between audio embeddings mirror those of their corresponding lyric embeddings.
• Figure 2: Architecture of the audio encoder $g_{\text{audio}}$. (a) Raw audio is first processed by the Whisper encoder to obtain hidden representations. (b) A [CLS] token is appended to aggregate frame-level features using an attention pooling mechanism. (c) A multi-layer perceptron projects the pooled representation into the lyrics-informed embedding space, yielding the final audio embedding $a_i$.
• Figure 3: Runtime and model size comparison. Average preprocessing and inference times are shown alongside model sizes for LIVI and baseline models. Error bars denote std across runs.