Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wav2CLIP: Learning Robust Audio Representations From CLIP

Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, Juan Pablo Bello

TL;DR

Wav2CLIP addresses the need for robust, data-efficient audio representations by distilling from CLIP into an audio encoder. The approach freezes the CLIP vision model and trains an audio encoder to predict CLIP image embeddings from video frames, optimizing with a cross-projection loss to align audio with the CLIP multimodal space. Evaluated across classification, retrieval, and audio captioning, Wav2CLIP matches or surpasses public audio representations while using roughly 10% of the annotated data and enabling zero-shot and cross-modal capabilities, including image generation from audio. The work demonstrates a scalable, lightweight pipeline that yields a multimodal, text-aligned audio embedding suitable for diverse downstream tasks and applications.

Abstract

We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ~10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.

Wav2CLIP: Learning Robust Audio Representations From CLIP

TL;DR

Wav2CLIP addresses the need for robust, data-efficient audio representations by distilling from CLIP into an audio encoder. The approach freezes the CLIP vision model and trains an audio encoder to predict CLIP image embeddings from video frames, optimizing with a cross-projection loss to align audio with the CLIP multimodal space. Evaluated across classification, retrieval, and audio captioning, Wav2CLIP matches or surpasses public audio representations while using roughly 10% of the annotated data and enabling zero-shot and cross-modal capabilities, including image generation from audio. The work demonstrates a scalable, lightweight pipeline that yields a multimodal, text-aligned audio embedding suitable for diverse downstream tasks and applications.

Abstract

We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ~10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.

Paper Structure

This paper contains 12 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experimental Design
  4. Pre-training: Distilling From CLIP
  5. Evaluation: Downstream Tasks
  6. Dataset and Tasks
  7. Baseline Comparisons
  8. Results and Discussions
  9. Classification, Retrieval, and Audio Captioning Tasks
  10. YamNet vs Wav2CLIP: Different Pretext Tasks
  11. Image Generation: Listening at a Glance
  12. Conclusion

Figures (5)

  • Figure 1: Contrastive Language-Image Pre-training (CLIP), and our two-stage approaches including pre-training and evaluation.
  • Figure 2: VGGSound audio classification results with different percentage (%) of training samples. Comparisons between Supervise, OpenL3, YamNet, and Wav2CLIP.
  • Figure 3: Confusion matrices of YamNet, Wav2CLIP, and the difference between them (more red on the diagonal and blue off-diagonal the better).
  • Figure 4: Examples of text (top) and audio (bottom) to image generation from UrbanSound8K dataset, corresponding labels in x-axis.
  • Figure 5: Images generated from individual tracks and mixtures of two songs in musdb18 musdb18-hq dataset. Top: Meaxic - Take A Step, Bottom: Atlantis Bound - It Was My Fault For Waiting.