Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data

Xuenan Xu, Zhiling Zhang, Zelin Zhou, Pingyue Zhang, Zeyu Xie, Mengyue Wu, Kenny Q. Zhu

TL;DR

This paper proposes to utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated and achieves state-of-the-art zero-shot classification performance on most datasets, suggesting the effectiveness of the synthetic data.

Abstract

Compared with ample visual-text pre-training research, few works explore audio-text pre-training, mostly due to the lack of sufficient parallel audio-text data. Most existing methods incorporate the visual modality as a pivot for audio-text pre-training, which inevitably induces data noise. In this paper, we propose to utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated. Furthermore, we propose caption generation under the guidance of AudioSet tags, leading to more accurate captions. With the above two improvements, we curate high-quality, large-scale parallel audio-text data, based on which we perform audio-text pre-training. We comprehensively demonstrate the performance of the pre-trained model on a series of downstream audio-related tasks, including single-modality tasks like audio classification and tagging, as well as cross-modal tasks consisting of audio-text retrieval and audio-based text generation. Experimental results indicate that our approach achieves state-of-the-art zero-shot classification performance on most datasets, suggesting the effectiveness of our synthetic data. The audio encoder also serves as an efficient pattern recognition model by fine-tuning it on audio-related tasks. Synthetic data and pre-trained models are available online.

BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data

TL;DR

This paper proposes to utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated and achieves state-of-the-art zero-shot classification performance on most datasets, suggesting the effectiveness of the synthetic data.

Abstract

Compared with ample visual-text pre-training research, few works explore audio-text pre-training, mostly due to the lack of sufficient parallel audio-text data. Most existing methods incorporate the visual modality as a pivot for audio-text pre-training, which inevitably induces data noise. In this paper, we propose to utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated. Furthermore, we propose caption generation under the guidance of AudioSet tags, leading to more accurate captions. With the above two improvements, we curate high-quality, large-scale parallel audio-text data, based on which we perform audio-text pre-training. We comprehensively demonstrate the performance of the pre-trained model on a series of downstream audio-related tasks, including single-modality tasks like audio classification and tagging, as well as cross-modal tasks consisting of audio-text retrieval and audio-based text generation. Experimental results indicate that our approach achieves state-of-the-art zero-shot classification performance on most datasets, suggesting the effectiveness of our synthetic data. The audio encoder also serves as an efficient pattern recognition model by fine-tuning it on audio-related tasks. Synthetic data and pre-trained models are available online.
Paper Structure (31 sections, 5 equations, 4 figures, 6 tables)

This paper contains 31 sections, 5 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Pre-training
  4. Audio-Language Pre-training
  5. Audio Event Recognition
  6. Audio Representation Learning
  7. Bootstrapping Language-Audio Data with AudioSet Tags
  8. Current Audio-Text Datasets
  9. Audio-Text Data Generation from AudioSet
  10. Audio-Text Pre-training
  11. Contrastive Pre-training
  12. Audio Encoder
  13. Text Encoder
  14. Experimental Setup
  15. Synthetic Audio-Text Data Generation
  16. ...and 16 more sections

Figures (4)

  • Figure 1: The illustration of the data expansion approach. "A", "E" and "C" denote audio, event tags and caption respectively.
  • Figure 2: The proposed audio captioning system with AudioSet tag guidance. The system generates caption based on both the input audio clip and the provided AudioSet tags.
  • Figure 3: An overview of our proposed language-audio pre-training approach. We use a tag-guided audio captioning model to generate audio-text data. Then we conduct contrastive learning similar to CLIP (dashed lines indicate the captioning model is not involved in the pre-training) in two stages. The pre-trained model can be transferred by zero-shot inference or fine-tuning.
  • Figure 4: An example of annotation errors in AudioSet. A woman is speaking in the audio clip while the corresponding event "Female speech, woman speaking" is not annotated.