Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning

Ilaria Manco, Justin Salamon, Oriol Nieto

TL;DR

Music-text contrastive models enable cross-modal music retrieval but often suffer under limited data and compute. This work systematically analyzes encoder backbones and training data, introducing Augment, Drop & Swap to diversify training captions and create hard negatives without added cost. The study finds data quality and curation to be more impactful than sheer dataset size, and demonstrates that the proposed augmentation techniques—Augmented View Dropout and TextSwap—consistently boost performance across architectures and languages. Multilingual retrieval with locked encoders and a listening study corroborate the practical value, achieving strong results without increasing data or compute requirements.

Abstract

Audio-text contrastive models have become a powerful approach in music representation learning. Despite their empirical success, however, little is known about the influence of key design choices on the quality of music-text representations learnt through this framework. In this work, we expose these design choices within the constraints of limited data and computation budgets, and establish a more solid understanding of their impact grounded in empirical observations along three axes: the choice of base encoders, the level of curation in training data, and the use of text augmentation. We find that data curation is the single most important factor for music-text contrastive training in resource-constrained scenarios. Motivated by this insight, we introduce two novel techniques, Augmented View Dropout and TextSwap, which increase the diversity and descriptiveness of text inputs seen in training. Through our experiments we demonstrate that these are effective at boosting performance across different pre-training regimes, model architectures, and downstream data distributions, without incurring higher computational costs or requiring additional training data.

Augment, Drop & Swap: Improving Diversity in LLM Captions for Efficient Music-Text Representation Learning

TL;DR

Music-text contrastive models enable cross-modal music retrieval but often suffer under limited data and compute. This work systematically analyzes encoder backbones and training data, introducing Augment, Drop & Swap to diversify training captions and create hard negatives without added cost. The study finds data quality and curation to be more impactful than sheer dataset size, and demonstrates that the proposed augmentation techniques—Augmented View Dropout and TextSwap—consistently boost performance across architectures and languages. Multilingual retrieval with locked encoders and a listening study corroborate the practical value, achieving strong results without increasing data or compute requirements.

Abstract

Audio-text contrastive models have become a powerful approach in music representation learning. Despite their empirical success, however, little is known about the influence of key design choices on the quality of music-text representations learnt through this framework. In this work, we expose these design choices within the constraints of limited data and computation budgets, and establish a more solid understanding of their impact grounded in empirical observations along three axes: the choice of base encoders, the level of curation in training data, and the use of text augmentation. We find that data curation is the single most important factor for music-text contrastive training in resource-constrained scenarios. Motivated by this insight, we introduce two novel techniques, Augmented View Dropout and TextSwap, which increase the diversity and descriptiveness of text inputs seen in training. Through our experiments we demonstrate that these are effective at boosting performance across different pre-training regimes, model architectures, and downstream data distributions, without incurring higher computational costs or requiring additional training data.
Paper Structure (14 sections, 5 figures, 4 tables)

This paper contains 14 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Studying the Design Space of Music-Text Embedding Models
  3. Our experimental approach
  4. Evaluation
  5. The role of encoder backbones
  6. Encoders: initialization and freezing
  7. Supporting retrieval in multiple languages
  8. The role of training data
  9. Tag-to-caption augmentation via LLMs
  10. Training data: size vs quality
  11. Improving diversity via text augmentations
  12. Augment, Drop & Swap
  13. Experiments
  14. Conclusions

Figures (5)

  • Figure 1: Overview of our approach. We study the role of encoders and data in music-text learning and propose a text augmentation pipeline, Augment, Drop & Swap, to increase data diversity and introduce hard negatives during training.
  • Figure 2: Retrieval performance (R@10) of different combinations of audio and text encoders compared through the lens of our DuET-MC framework.
  • Figure 3: The effect of varying $\boldsymbol{p_{cap}}$, the probability of swapping tags with captions. On the y-axis, we show the relative change in performance compared to $p_{cap}=0$.
  • Figure 4: Retrieval performance across models trained on datasets that differ in size and annotation quality.
  • Figure :