Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Separate What You Describe: Language-Queried Audio Source Separation

Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D. Plumbley, Wenwu Wang

TL;DR

Language-queried audio source separation (LASS) introduces a task where a target source is extracted from a mixture based on a natural language description. The proposed LASS-Net jointly encodes the language query with a Transformer-based QueryNet (BERT) and conditions a ResUNet-based SeparationNet via FiLM to produce a target-specific spectrogram mask, trained with the MAE on magnitude spectra. A dataset derived from AudioCaps demonstrates the approach, showing that LASS-Net significantly outperforms a tag-based baseline and generalizes to diverse human descriptions. This work enables flexible, language-driven audio editing and retrieval, and provides a foundation for real-world language-conditioned audio separation systems.

Abstract

In this paper, we introduce the task of language-queried audio source separation (LASS), which aims to separate a target source from an audio mixture based on a natural language query of the target source (e.g., "a man tells a joke followed by people laughing"). A unique challenge in LASS is associated with the complexity of natural language description and its relation with the audio sources. To address this issue, we proposed LASS-Net, an end-to-end neural network that is learned to jointly process acoustic and linguistic information, and separate the target source that is consistent with the language query from an audio mixture. We evaluate the performance of our proposed system with a dataset created from the AudioCaps dataset. Experimental results show that LASS-Net achieves considerable improvements over baseline methods. Furthermore, we observe that LASS-Net achieves promising generalization results when using diverse human-annotated descriptions as queries, indicating its potential use in real-world scenarios. The separated audio samples and source code are available at https://liuxubo717.github.io/LASS-demopage.

Separate What You Describe: Language-Queried Audio Source Separation

TL;DR

Language-queried audio source separation (LASS) introduces a task where a target source is extracted from a mixture based on a natural language description. The proposed LASS-Net jointly encodes the language query with a Transformer-based QueryNet (BERT) and conditions a ResUNet-based SeparationNet via FiLM to produce a target-specific spectrogram mask, trained with the MAE on magnitude spectra. A dataset derived from AudioCaps demonstrates the approach, showing that LASS-Net significantly outperforms a tag-based baseline and generalizes to diverse human descriptions. This work enables flexible, language-driven audio editing and retrieval, and provides a foundation for real-world language-conditioned audio separation systems.

Abstract

In this paper, we introduce the task of language-queried audio source separation (LASS), which aims to separate a target source from an audio mixture based on a natural language query of the target source (e.g., "a man tells a joke followed by people laughing"). A unique challenge in LASS is associated with the complexity of natural language description and its relation with the audio sources. To address this issue, we proposed LASS-Net, an end-to-end neural network that is learned to jointly process acoustic and linguistic information, and separate the target source that is consistent with the language query from an audio mixture. We evaluate the performance of our proposed system with a dataset created from the AudioCaps dataset. Experimental results show that LASS-Net achieves considerable improvements over baseline methods. Furthermore, we observe that LASS-Net achieves promising generalization results when using diverse human-annotated descriptions as queries, indicating its potential use in real-world scenarios. The separated audio samples and source code are available at https://liuxubo717.github.io/LASS-demopage.
Paper Structure (20 sections, 6 equations, 3 figures, 2 tables)

This paper contains 20 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Universal sound separation
  4. Target source extraction
  5. Automated audio captioning
  6. Proposed Approach
  7. LASS-Net
  8. Query network
  9. Separation network
  10. Dataset
  11. Experiments and Results
  12. Data processing
  13. Baseline system
  14. Training procedure
  15. Evaluation metrics
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Framework of our proposed LASS-Net.
  • Figure 2: Box plots of the SDR scores on two experiments: (a) Evaluation on LASS-Test (b) Evaluation on LASS-Sub-Test. Red solid line represents the median value.
  • Figure 3: Spectrogram visualizations of (a) an audio mixture, (b) The ground truth target source, (c) The separated source queried by AudioCaps description: "applauding followed by people singing and a tambourine", and (d) The separated source queried by our collected human description: "a show start with audience applauding and then singing".