Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Separate Anything You Describe

Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, Wenwu Wang

TL;DR

AudioSep presents a foundation model for open-domain language-queried audio source separation, leveraging large-scale multimodal data and a two-component architecture (QueryNet and SeparationNet) to achieve strong zero-shot generalization across diverse tasks. By integrating CLAP-based language embeddings with a ResUNet separation backbone through FiLM conditioning, AudioSep scales beyond fixed-label or instrument-specific separation to hundreds of sound classes. The study demonstrates strong performance on seen and unseen datasets, outperforms state-of-the-art LASS and audio-queried baselines, and provides a comprehensive evaluation benchmark with substantial ablations on multimodal supervision and text-query variations. The work suggests promising directions for unsupervised learning and future extension to audio-visual and speaker-aware separation, with public code and benchmarks to foster further research.

Abstract

Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA). LASS aims to separate a target sound from an audio mixture given a natural language query, which provides a natural and scalable interface for digital audio applications. Recent works on LASS, despite attaining promising separation performance on specific sources (e.g., musical instruments, limited classes of audio events), are unable to separate audio concepts in the open domain. In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries. We train AudioSep on large-scale multimodal datasets and extensively evaluate its capabilities on numerous tasks including audio event separation, musical instrument separation, and speech enhancement. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability using audio captions or text labels as queries, substantially outperforming previous audio-queried and language-queried sound separation models. For reproducibility of this work, we will release the source code, evaluation benchmark and pre-trained model at: https://github.com/Audio-AGI/AudioSep.

Separate Anything You Describe

TL;DR

AudioSep presents a foundation model for open-domain language-queried audio source separation, leveraging large-scale multimodal data and a two-component architecture (QueryNet and SeparationNet) to achieve strong zero-shot generalization across diverse tasks. By integrating CLAP-based language embeddings with a ResUNet separation backbone through FiLM conditioning, AudioSep scales beyond fixed-label or instrument-specific separation to hundreds of sound classes. The study demonstrates strong performance on seen and unseen datasets, outperforms state-of-the-art LASS and audio-queried baselines, and provides a comprehensive evaluation benchmark with substantial ablations on multimodal supervision and text-query variations. The work suggests promising directions for unsupervised learning and future extension to audio-visual and speaker-aware separation, with public code and benchmarks to foster further research.

Abstract

Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA). LASS aims to separate a target sound from an audio mixture given a natural language query, which provides a natural and scalable interface for digital audio applications. Recent works on LASS, despite attaining promising separation performance on specific sources (e.g., musical instruments, limited classes of audio events), are unable to separate audio concepts in the open domain. In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries. We train AudioSep on large-scale multimodal datasets and extensively evaluate its capabilities on numerous tasks including audio event separation, musical instrument separation, and speech enhancement. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability using audio captions or text labels as queries, substantially outperforming previous audio-queried and language-queried sound separation models. For reproducibility of this work, we will release the source code, evaluation benchmark and pre-trained model at: https://github.com/Audio-AGI/AudioSep.
Paper Structure (46 sections, 5 equations, 4 figures, 11 tables)

This paper contains 46 sections, 5 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio Source separation
  4. Universal sound separation
  5. Query-based sound separation
  6. Audio-visual sound separation
  7. Audio-queried sound separation
  8. Label-queried sound separation
  9. Language-queried audio source separation
  10. Multimodal audio-language learning
  11. Method
  12. QueryNet
  13. SeparationNet
  14. Loss and training
  15. Connections to the prior works
  16. ...and 31 more sections

Figures (4)

  • Figure 1: Framework of AudioSep. AudioSep has two key components: a QueryNet and a SeparationNet. The QueryNet is the text encoder of CLIP clip or CLAP clap model. The SeparationNet is a frequency-domain ResUNet kong2021decouplingkong2023universal model.
  • Figure 2: Visualization of separation results obtained by AudioSep.
  • Figure 3: Case studies of (a) an audio mixture; (b) the ground truth target source; (c) the separated source queried by AudioCaps's original caption: “People laugh followed by people singing while music plays"; (d) the separated source queried by our reannotated caption: “A music show is presenting to the public". Results are obtained using the AudioSep-CLAP-TR1.0 model.
  • Figure 4: An example of performing separation using AudioSep with an invalid query: "Separate Anything You Describe".