Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AudioCLIP: Extending CLIP to Image, Text and Audio

Andrey Guzhov, Federico Raue, Jörn Hees, Andreas Dengel

TL;DR

AudioCLIP extends CLIP from two to three modalities by integrating the ESResNeXt audio model, training on AudioSet to align text, image, and audio in a shared embedding space. The tri-modal framework introduces additional loss terms for text-audio and image-audio alignment, enabling zero-shot classification and cross-modal querying across all modality combinations. The approach achieves state-of-the-art results on environmental sound benchmarks (UrbanSound8K and ESC-50) and establishes strong zero-shot baselines, while also enabling retrieval across modalities. The work demonstrates the feasibility and benefits of tri-modal contrastive learning for robust, generalizable audio representations and multimodal retrieval tasks, with reproducibility via released code and models.

Abstract

In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models. In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively). Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.

AudioCLIP: Extending CLIP to Image, Text and Audio

TL;DR

AudioCLIP extends CLIP from two to three modalities by integrating the ESResNeXt audio model, training on AudioSet to align text, image, and audio in a shared embedding space. The tri-modal framework introduces additional loss terms for text-audio and image-audio alignment, enabling zero-shot classification and cross-modal querying across all modality combinations. The approach achieves state-of-the-art results on environmental sound benchmarks (UrbanSound8K and ESC-50) and establishes strong zero-shot baselines, while also enabling retrieval across modalities. The work demonstrates the feasibility and benefits of tri-modal contrastive learning for robust, generalizable audio representations and multimodal retrieval tasks, with reproducibility via released code and models.

Abstract

In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models. In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion. AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively). Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.

Paper Structure

This paper contains 23 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model
  4. CLIP
  5. ESResNeXt
  6. Hybrid Model -- AudioCLIP
  7. Experimental Setup
  8. Datasets
  9. Data Augmentation
  10. Training
  11. Audio-Head Pre-Training
  12. AudioCLIP Training
  13. Audio-Head Fine-Tuning
  14. Hyper-Parameters
  15. Performance Evaluation
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Overview of the proposed AudioCLIP model. On the left, the workflow of the text-image-model CLIP is shown. Performing joint training of the text- and image-heads, CLIP learns to align representations of the same concept in a shared multimodal embedding space. On the right, the audio-model ESResNeXT is shown. Here, the added audible modality interacts with two others, enabling the model to handle 3 modalities simultaneously.