Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Audio-Only Data for Text-Queried Target Sound Extraction

Kohei Saijo, Janek Ebbers, François G. Germain, Sameer Khurana, Gordon Wichern, Jonathan Le Roux

TL;DR

The paper tackles the data scarcity of text-audio pairs in text-queried target sound extraction by leveraging audio-only data. It uses CLAP to obtain audio embeddings during training and switches to text embeddings at inference, but faces a modality gap that can cause overfitting to audio-specific features. Through simple embedding-manipulation techniques, notably embedding dropout, the authors show audio-only training can match or exceed text-based training and even improve robustness to out-of-domain data. The findings offer a scalable pathway to incorporate vast audio-only data into text-queried TSE and suggest further gains from larger-scale audio datasets.

Abstract

The goal of text-queried target sound extraction (TSE) is to extract from a mixture a sound source specified with a natural-language caption. While it is preferable to have access to large-scale text-audio pairs to address a variety of text prompts, the limited number of available high-quality text-audio pairs hinders the data scaling. To this end, this work explores how to leverage audio-only data without any captions for the text-queried TSE task to potentially scale up the data amount. A straightforward way to do so is to use a joint audio-text embedding model, such as the contrastive language-audio pre-training (CLAP) model, as a query encoder and train a TSE model using audio embeddings obtained from the ground-truth audio. The TSE model can then accept text queries at inference time by switching to the text encoder. While this approach should work if the audio and text embedding spaces in CLAP were well aligned, in practice, the embeddings have domain-specific information that causes the TSE model to overfit to audio queries. We investigate several methods to avoid overfitting and show that simple embedding-manipulation methods such as dropout can effectively alleviate this issue. Extensive experiments demonstrate that using audio-only data with embedding dropout is as effective as using text captions during training, and audio-only data can be effectively leveraged to improve text-queried TSE models.

Leveraging Audio-Only Data for Text-Queried Target Sound Extraction

TL;DR

The paper tackles the data scarcity of text-audio pairs in text-queried target sound extraction by leveraging audio-only data. It uses CLAP to obtain audio embeddings during training and switches to text embeddings at inference, but faces a modality gap that can cause overfitting to audio-specific features. Through simple embedding-manipulation techniques, notably embedding dropout, the authors show audio-only training can match or exceed text-based training and even improve robustness to out-of-domain data. The findings offer a scalable pathway to incorporate vast audio-only data into text-queried TSE and suggest further gains from larger-scale audio datasets.

Abstract

The goal of text-queried target sound extraction (TSE) is to extract from a mixture a sound source specified with a natural-language caption. While it is preferable to have access to large-scale text-audio pairs to address a variety of text prompts, the limited number of available high-quality text-audio pairs hinders the data scaling. To this end, this work explores how to leverage audio-only data without any captions for the text-queried TSE task to potentially scale up the data amount. A straightforward way to do so is to use a joint audio-text embedding model, such as the contrastive language-audio pre-training (CLAP) model, as a query encoder and train a TSE model using audio embeddings obtained from the ground-truth audio. The TSE model can then accept text queries at inference time by switching to the text encoder. While this approach should work if the audio and text embedding spaces in CLAP were well aligned, in practice, the embeddings have domain-specific information that causes the TSE model to overfit to audio queries. We investigate several methods to avoid overfitting and show that simple embedding-manipulation methods such as dropout can effectively alleviate this issue. Extensive experiments demonstrate that using audio-only data with embedding dropout is as effective as using text captions during training, and audio-only data can be effectively leveraged to improve text-queried TSE models.
Paper Structure (13 sections, 2 equations, 1 figure, 3 tables)

This paper contains 13 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Audio-queried training for text-queried TSE
  3. Problem setup
  4. Modality gap in CLAP
  5. Methods for alleviating modality gap
  6. Related work
  7. Experiments
  8. Datasets
  9. Models
  10. Training/evaluation details
  11. Main results
  12. Ablation study
  13. Conclusion

Figures (1)

  • Figure 1: Illustration of text-queried-TSE training with audio-only data. Audio embeddings extracted from ground-truth audio by a CLAP audio encoder are used to condition the extraction model during training, while text embeddings are used during inference. To prevent the extraction model from overfitting to audio-embedding-specific features, we apply a simple manipulation (e.g., dimension dropout, PCA, etc.) to the embeddings.