Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation

Junda Wu, Warren Li, Zachary Novack, Amit Namburi, Carol Chen, Julian McAuley

TL;DR

A novel contrastive learning architecture that fuses language representations with structured audio representations by segmenting each song into clips and extracting their embeddings is proposed, allowing the model to automatically weigh and enhance the final fusion score for improved contrastive alignment.

Abstract

Modeling temporal characteristics plays a significant role in the representation learning of audio waveform. We propose Contrastive Long-form Language-Audio Pretraining (\textbf{CoLLAP}) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words), while enabling contrastive learning across modalities and temporal dynamics. Leveraging recent Music-LLMs to generate long-form music captions for full-length songs, augmented with musical temporal structures, we collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds. We propose a novel contrastive learning architecture that fuses language representations with structured audio representations by segmenting each song into clips and extracting their embeddings. With an attention mechanism, we capture multimodal temporal correlations, allowing the model to automatically weigh and enhance the final fusion score for improved contrastive alignment. Finally, we develop two variants of the CoLLAP model with different types of backbone language models. Through comprehensive experiments on multiple long-form music-text retrieval datasets, we demonstrate consistent performance improvement in retrieval accuracy compared with baselines. We also show the pretrained CoLLAP models can be transferred to various music information retrieval tasks, with heterogeneous long-form multimodal contexts.

CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation

TL;DR

A novel contrastive learning architecture that fuses language representations with structured audio representations by segmenting each song into clips and extracting their embeddings is proposed, allowing the model to automatically weigh and enhance the final fusion score for improved contrastive alignment.

Abstract

Modeling temporal characteristics plays a significant role in the representation learning of audio waveform. We propose Contrastive Long-form Language-Audio Pretraining (\textbf{CoLLAP}) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words), while enabling contrastive learning across modalities and temporal dynamics. Leveraging recent Music-LLMs to generate long-form music captions for full-length songs, augmented with musical temporal structures, we collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds. We propose a novel contrastive learning architecture that fuses language representations with structured audio representations by segmenting each song into clips and extracting their embeddings. With an attention mechanism, we capture multimodal temporal correlations, allowing the model to automatically weigh and enhance the final fusion score for improved contrastive alignment. Finally, we develop two variants of the CoLLAP model with different types of backbone language models. Through comprehensive experiments on multiple long-form music-text retrieval datasets, we demonstrate consistent performance improvement in retrieval accuracy compared with baselines. We also show the pretrained CoLLAP models can be transferred to various music information retrieval tasks, with heterogeneous long-form multimodal contexts.
Paper Structure (13 sections, 10 equations, 2 figures, 4 tables)

This paper contains 13 sections, 10 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. CoLLAP: Model Design and Learning
  3. Text and Dynamic Audio Encoders
  4. Multimodal and Temporal Attention Augmentation
  5. Temporal Attention Fused Contrastrive Learning
  6. Long-form and Structural-aware Text-audio Retrieval Dataset
  7. Experiments
  8. Implementation Details
  9. Datasets and Baselines
  10. Long-form Text-audio Retrieval
  11. Zero-shot Transcript-speech Retrieval
  12. Zero-shot Wiki-music Retrieval
  13. Conclusion

Figures (2)

  • Figure 1: Comparison of conventional CLAP (Figure \ref{['fig:1-1']}) and our proposed CoLLAP (Figure \ref{['fig:1-2']}).
  • Figure 2: The model overview of CoLLAP. The input of backbone language models is musical structural augmented texts, while audio waveform is encoded by the dual-feature extractor of Beats and Whisper models. The encoded multimodal features are used for the calculation of temporal and kernel-wise attentions before computing contrastive learning loss.