Language-Queried Target Sound Extraction Without Parallel Training Data

TL;DR

The paper tackles language-queried target sound extraction (TSE) without relying on parallel audio-text data. It leverages contrastive language-audio pre-training (CLAP) to create modality-aligned embeddings and introduces a retrieval-augmented strategy that builds an embedding cache from large-language-model-produced audio captions, enabling training-time retrieval of text embeddings that match the target audio embedding. This approach addresses training-testing modality gaps and information leakage, leading to consistent improvements over state-of-the-art and strong generalization, particularly when scaling to large unlabeled datasets. The method significantly reduces labeling costs while enabling scalable, language-conditioned sound extraction across diverse benchmarks, achieving notable SDR gains and robust cross-dataset performance.

Abstract

Language-queried target sound extraction (TSE) aims to extract specific sounds from mixtures based on language queries. Traditional fully-supervised training schemes require extensively annotated parallel audio-text data, which are labor-intensive. We introduce a parallel-data-free training scheme, requiring only unlabelled audio clips for TSE model training by utilizing the contrastive language-audio pre-trained model (CLAP). In a vanilla parallel-data-free training stage, target audio is encoded using the pre-trained CLAP audio encoder to form a condition embedding, while during testing, user language queries are encoded by CLAP text encoder as the condition embedding. This vanilla approach assumes perfect alignment between text and audio embeddings, which is unrealistic. Two major challenges arise from training-testing mismatch: the persistent modality gap between text and audio and the risk of overfitting due to the exposure of rich acoustic details in target audio embedding during training. To address this, we propose a retrieval-augmented strategy. Specifically, we create an embedding cache using audio captions generated by a large language model (LLM). During training, target audio embeddings retrieve text embeddings from this cache to use as condition embeddings, ensuring consistent modalities between training and testing and eliminating information leakage. Extensive experiment results show that our retrieval-augmented approach achieves consistent and notable performance improvements over existing state-of-the-art with better generalizability.

Figures (3)

• Figure 1: Overview of our proposed retrieval-augmented parallel-data-free training scheme for language-queried target sound extraction.
• Figure 2: Effect of Gaussian perturbation strength.
• Figure 3: Effect of LLM-produced captions.