Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Effective Negation Modeling in Joint Audio-Text Models for Music

Yannis Vasilakis, Rachel Bittner, Johan Pauwels

TL;DR

Negation remains a core gap in joint audio-text models for music retrieval. The authors train CLAP from scratch on MSD with LP-MusicCaps-MSD and introduce two negation-focused strategies: Negation insert text augmentation and a dissimilarity-based loss that explicitly separates negated captions from originals. They evaluate negation through retrieval and binary classification tasks, showing that each method, alone or together, enhances negation signaling while largely preserving retrieval performance. The study provides practical negation modeling techniques, a fair baseline CLAP model trained on public data, and evaluation protocols to guide future work on negation in multimodal music understanding.

Abstract

Joint audio-text models are widely used for music retrieval, yet they struggle with semantic phenomena such as negation. Negation is fundamental for distinguishing the absence (or presence) of musical elements (e.g., "with vocals" vs. "without vocals"), but current systems fail to represent this reliably. In this work, we investigate and mitigate this limitation by training CLAP models from scratch on the Million Song Dataset with LP-MusicCaps-MSD captions. We introduce negation through text augmentation and a dissimilarity-based contrastive loss, designed to explicitly separate original and negated captions in the joint embedding space. To evaluate progress, we propose two protocols that frame negation modeling as retrieval and binary classification tasks. Experiments demonstrate that both methods, individually and combined, improve negation handling while largely preserving retrieval performance.

Towards Effective Negation Modeling in Joint Audio-Text Models for Music

TL;DR

Negation remains a core gap in joint audio-text models for music retrieval. The authors train CLAP from scratch on MSD with LP-MusicCaps-MSD and introduce two negation-focused strategies: Negation insert text augmentation and a dissimilarity-based loss that explicitly separates negated captions from originals. They evaluate negation through retrieval and binary classification tasks, showing that each method, alone or together, enhances negation signaling while largely preserving retrieval performance. The study provides practical negation modeling techniques, a fair baseline CLAP model trained on public data, and evaluation protocols to guide future work on negation in multimodal music understanding.

Abstract

Joint audio-text models are widely used for music retrieval, yet they struggle with semantic phenomena such as negation. Negation is fundamental for distinguishing the absence (or presence) of musical elements (e.g., "with vocals" vs. "without vocals"), but current systems fail to represent this reliably. In this work, we investigate and mitigate this limitation by training CLAP models from scratch on the Million Song Dataset with LP-MusicCaps-MSD captions. We introduce negation through text augmentation and a dissimilarity-based contrastive loss, designed to explicitly separate original and negated captions in the joint embedding space. To evaluate progress, we propose two protocols that frame negation modeling as retrieval and binary classification tasks. Experiments demonstrate that both methods, individually and combined, improve negation handling while largely preserving retrieval performance.
Paper Structure (14 sections, 2 equations, 4 figures, 2 tables)

This paper contains 14 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Enhancing negation modeling
  3. Negation insert - negation text augmentation
  4. Dissimilarity term - adding a loss term
  5. Evaluation of negation modeling
  6. Negation as retrieval
  7. Negation as binary classification
  8. Experiments
  9. Dataset and models
  10. Impact of text augmentation on retrieval metrics
  11. Impact of Dissimilarity term on retrieval metrics
  12. Impact of the combination of proposed methods on retrieval metrics
  13. Evaluation of negation as a binary classification task
  14. Conclusion

Figures (4)

  • Figure 1: Sensitivity analysis of text augmentation (text aug) using different augmentation probabilities.
  • Figure 2: Sensitivity analysis of Dissimilarity term (loss term) using different term weights $k$.
  • Figure 3: Sensitivity analysis of combining text aug and loss term (combo) using different term weights $k$.
  • Figure 4: Binary classification evaluation of 5 CLAP variants (see section \ref{['subsec:negation_as_binary_classification']}). Arrow signifies experiments where the loss term weight increases. We highlight that text aug is combo with $k=0$.