Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controllable Embedding Transformation for Mood-Guided Music Retrieval

Julia Wilkins, Jaehun Kim, Matthew E. P. Davies, Juan Pablo Bello, Matthew C. McCallum

TL;DR

This work tackles mood-controlled music retrieval by learning a transformation in the embedding space that moves a seed track toward a target mood while preserving non-mood attributes like genre and instrumentation. It introduces a novel nearest-neighbor sampling scheme to generate proxy mood-target embeddings and trains a lightweight translator in the MULE embedding space using a joint objective with three complementary losses. Across a large proprietary dataset and MTG-Jamendo, the method achieves strong mood transformation while substantially preserving seed information, outperforming training-free baselines and approaching Oracle upper bounds. The approach generalizes to zero-shot scenarios and suggests promising directions for extending controllability to multi-mood settings and cross-modal guidance. This work establishes controllable embedding transformation as a viable paradigm for scalable, attribute-guided music retrieval.

Abstract

Music representations are the backbone of modern recommendation systems, powering playlist generation, similarity search, and personalized discovery. Yet most embeddings offer little control for adjusting a single musical attribute, e.g., changing only the mood of a track while preserving its genre or instrumentation. In this work, we address the problem of controllable music retrieval through embedding-based transformation, where the objective is to retrieve songs that remain similar to a seed track but are modified along one chosen dimension. We propose a novel framework for mood-guided music embedding transformation, which learns a mapping from a seed audio embedding to a target embedding guided by mood labels, while preserving other musical attributes. Because mood cannot be directly altered in the seed audio, we introduce a sampling mechanism that retrieves proxy targets to balance diversity with similarity to the seed. We train a lightweight translation model using this sampling strategy and introduce a novel joint objective that encourages transformation and information preservation. Extensive experiments on two datasets show strong mood transformation performance while retaining genre and instrumentation far better than training-free baselines, establishing controllable embedding transformation as a promising paradigm for personalized music retrieval.

Controllable Embedding Transformation for Mood-Guided Music Retrieval

TL;DR

This work tackles mood-controlled music retrieval by learning a transformation in the embedding space that moves a seed track toward a target mood while preserving non-mood attributes like genre and instrumentation. It introduces a novel nearest-neighbor sampling scheme to generate proxy mood-target embeddings and trains a lightweight translator in the MULE embedding space using a joint objective with three complementary losses. Across a large proprietary dataset and MTG-Jamendo, the method achieves strong mood transformation while substantially preserving seed information, outperforming training-free baselines and approaching Oracle upper bounds. The approach generalizes to zero-shot scenarios and suggests promising directions for extending controllability to multi-mood settings and cross-modal guidance. This work establishes controllable embedding transformation as a viable paradigm for scalable, attribute-guided music retrieval.

Abstract

Music representations are the backbone of modern recommendation systems, powering playlist generation, similarity search, and personalized discovery. Yet most embeddings offer little control for adjusting a single musical attribute, e.g., changing only the mood of a track while preserving its genre or instrumentation. In this work, we address the problem of controllable music retrieval through embedding-based transformation, where the objective is to retrieve songs that remain similar to a seed track but are modified along one chosen dimension. We propose a novel framework for mood-guided music embedding transformation, which learns a mapping from a seed audio embedding to a target embedding guided by mood labels, while preserving other musical attributes. Because mood cannot be directly altered in the seed audio, we introduce a sampling mechanism that retrieves proxy targets to balance diversity with similarity to the seed. We train a lightweight translation model using this sampling strategy and introduce a novel joint objective that encourages transformation and information preservation. Extensive experiments on two datasets show strong mood transformation performance while retaining genre and instrumentation far better than training-free baselines, establishing controllable embedding transformation as a promising paradigm for personalized music retrieval.
Paper Structure (13 sections, 3 equations, 2 figures, 1 table)

This paper contains 13 sections, 3 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Mood-Guided Embedding Transformation Framework
  4. Objective Design
  5. Experimental Design
  6. Datasets
  7. Training Setup
  8. Evaluation Metrics
  9. Baseline Methods
  10. Results and Discussion
  11. Core Results
  12. Loss Ablation Study
  13. Conclusion

Figures (2)

  • Figure 1: A MULE embedding of a seed song is transformed, guided by a target mood label, into an embedding that can retrieve similar songs modified along the mood dimension. We evaluate in terms of mood transformation and seed information preservation.
  • Figure 2: Loss ablation using the large-scale dataset and MTG-Jamendo. Results shown are in percentage-points (pp) difference from the random baseline, scaled by $100$. For both datasets, the random baseline for Mood P@1 is $0.25$. For Genre P@1, the baseline is $0.05$ for the large-scale dataset and $0.011$ for MTG-Jamendo.