Musical Word Embedding for Music Tagging and Retrieval

TL;DR

Musical Word Embedding (MWE) addresses the gap between general text representations and musical semantics by training on a spectrum of corpora from general to music-specific. It integrates MWE into an audio–word joint embedding framework via a triplet network with multi-source supervision (tags, artist, track) and demonstrates improved performance across tag ranking, tagging, and retrieval tasks, including zero-shot scenarios on MSD and MTG-Jamendo. Key findings show that highly specific terms (track) excel in retrieval while broader terms (tag) aid tagging, and that multi-prototype supervision balances performance across tasks. The approach yields robust, scalable music tagging and retrieval with practical implications for search and recommendation, while noting limitations such as language scope and opportunities for multilingual extension.

Abstract

Word embedding has become an essential means for text-based information retrieval. Typically, word embeddings are learned from large quantities of general and unstructured text data. However, in the domain of music, the word embedding may have difficulty understanding musical contexts or recognizing music-related entities like artists and tracks. To address this issue, we propose a new approach called Musical Word Embedding (MWE), which involves learning from various types of texts, including both everyday and music-related vocabulary. We integrate MWE into an audio-word joint representation framework for tagging and retrieving music, using words like tag, artist, and track that have different levels of musical specificity. Our experiments show that using a more specific musical word like track results in better retrieval performance, while using a less specific term like tag leads to better tagging performance. To balance this compromise, we suggest multi-prototype training that uses words with different levels of musical specificity jointly. We evaluate both word embedding and audio-word joint embedding on four tasks (tag rank prediction, music tagging, query-by-tag, and query-by-track) across two datasets (Million Song Dataset and MTG-Jamendo). Our findings show that the suggested MWE is more efficient and robust than the conventional word embedding.

Figures (5)

• Figure 1: An illustration of training the musical word embedding. Word embedding vectors within a context window are shown with the same color (pink or blue).
• Figure 2: An illustration of the losses for supervision in the audio-word metric learning. Embedding vectors associated with the anchor are colored in pink if they are of the same class of the anchor (i.e., positive) and in blue otherwise (i.e., negative).
• Figure 3: A summary of evaluation tasks for word embeddings and audio-word joint embeddings. The embedding vectors associated with the input (or query) are colored in pink if they are of the same class (i.e., positive) or in blue otherwise (i.e., negative).
• Figure 4: Comparison of tag cosine similarity between word embedding models.
• Figure 5: The UMAP embedding visualization of word embedding (first row) and audio-word joint embedding (second row). Each color represents a similar semantic cluster. We note that (d) is a proposed musical word embedding.