Cacophony: An Improved Contrastive Audio-Text Model

TL;DR

A large-scale audio-text dataset containing 13,000 hours of text-labeled audio is crafted, using pretrained language models to process noisy text descriptions and automatic captioning to obtain text descriptions for unlabeled audio samples and achieves state-of-the-art performance on audio-text retrieval tasks.

Abstract

Despite recent advancements, audio-text models still lag behind their image-text counterparts in scale and performance. In this paper, we propose to improve both the data scale and the training procedure of audio-text contrastive models. Specifically, we craft a large-scale audio-text dataset containing 13,000 hours of text-labeled audio, using pretrained language models to process noisy text descriptions and automatic captioning to obtain text descriptions for unlabeled audio samples. We first train on audio-only data with a masked autoencoder (MAE) objective, which allows us to benefit from the scalability of unlabeled audio datasets. We then train a contrastive model with an auxiliary captioning objective with the audio encoder initialized from the MAE model. Our final model, which we name Cacophony, achieves state-of-the-art performance on audio-text retrieval tasks, and exhibits competitive results on the HEAR benchmark and other downstream tasks such as zero-shot classification.

Figures (6)

• Figure 1: A bar graph of the number of samples from commonly-used public audio datasets.
• Figure 2: Overview of our proposed dataset creation pipeline. Left: We process text descriptions based on the dataset quality. It aims to automatically clean and generate audio captions while maintaining consistency between the audio content and the textual descriptions. Middle: For datasets with raw, noisy descriptions, we use large language models, GPT-3 and T5-XXL, to clean the information that is irrelevant to the sound. Right: The detailed process of description cleaning using GPT-3 based on few-shot prompting is outlined as follows: Freesound raw inputs are highlighted in blue, sample inputs provided by a human annotator are marked in red, and the output text generated by GPT-3 is shown in bold red.
• Figure 3: The proposed system consists of a two-stage training process, as illustrated in the system diagram: Left (Stage 1): Audio-MAE training: This stage is conducted on our collected dataset. After this stage, the audio decoder is discarded, and the encoder is retained to initialize the audio encoder for the second stage of training. Right (Stage 2): Contrastive-captioning training: This stage involves training three components – the audio encoder, the text encoder, and the text decoder. The second stage is dedicated to achieving a contrastive-captioning objective, aligning and fine-tuning the interaction between the audio and text components.
• Figure 4: Contrastive-Captioning objective on the validation set during training, comparing scenarios with and without the application of SAM. For the cases with SAM optimization, we employ various neighborhood sizes as determined by the hyperparameter $\rho$.
• Figure 5: Comparison of classification accuracy ($\%$) on HEAR benchmark. Evaluation scores are stable across tasks, with a median 95$\%$ confidence interval of 0.25$\%$ with hear-eval-kit model-selection strategy. Comparisons of audio branches from contrastive language-audio embeddings.