Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval

TL;DR

TTMR++ tackles the Text-to-Music Retrieval problem by extending joint audio-text embeddings with rich text descriptions and metadata. It introduces a finetuned Large Language Model to generate pseudo-captions and leverages a music knowledge graph to create metadata-driven descriptions, all trained under a unified cross-modal framework. The method demonstrates state-of-the-art performance across caption-, tag-, track-, and artist-based retrieval tasks, showing the value of combining content semantics, metadata, and relative similarity signals. This approach enables more flexible and scalable music search in large databases, including similarity-based queries like 'music similar to ...' while leveraging existing tagging and metadata resources.

Abstract

Text-to-Music Retrieval, finding music based on a given natural language query, plays a pivotal role in content discovery within extensive music databases. To address this challenge, prior research has predominantly focused on a joint embedding of music audio and text, utilizing it to retrieve music tracks that exactly match descriptive queries related to musical attributes (i.e. genre, instrument) and contextual elements (i.e. mood, theme). However, users also articulate a need to explore music that shares similarities with their favorite tracks or artists, such as \textit{I need a similar track to Superstition by Stevie Wonder}. To address these concerns, this paper proposes an improved Text-to-Music Retrieval model, denoted as TTMR++, which utilizes rich text descriptions generated with a finetuned large language model and metadata. To accomplish this, we obtained various types of seed text from several existing music tag and caption datasets and a knowledge graph dataset of artists and tracks. The experimental results show the effectiveness of TTMR++ in comparison to state-of-the-art music-text joint embedding models through a comprehensive evaluation involving various musical text queries.

Figures (4)

• Figure 1: An illustration of text-to-music retrieval scenario with content, track, and artist queries.
• Figure 2: Music Audio and Text Contrastive Learning Framework.
• Figure 3: Connections between Artist and Track Entity.
• Figure 4: Tag-to-Caption Generation Results for LLaMA-7B, Finetuned LLaMA-7B (Ours), and GPT3.5-175B+.