Text-Queried Audio Source Separation via Hierarchical Modeling

TL;DR

This work introduces HSM-TSS, a hierarchical framework for text-queried audio source separation that decouples global semantic guidance from local, structure-preserving reconstruction. By employing a global semantic stage via Q-Audio and a local semantic stage via AudioMAE, followed by a semantic-to-acoustic decoder, the approach achieves state-of-the-art results with data-efficient training. The model demonstrates strong cross-modal alignment, robust performance in challenging mixtures, and impressive zero-shot generalization, aided by a bidirectional instruction framework and an autoregressive neural codec. Overall, the method advances open-domain, text-driven audio manipulation with practical implications for flexible audio editing and retrieval tasks.

Abstract

Target audio source separation with natural language queries presents a promising paradigm for extracting arbitrary audio events through arbitrary text descriptions. Existing methods mainly face two challenges, the difficulty in jointly modeling acoustic-textual alignment and semantic-aware separation within a blindly-learned single-stage architecture, and the reliance on large-scale accurately-labeled training data to compensate for inefficient cross-modal learning and separation. To address these challenges, we propose a hierarchical decomposition framework, HSM-TSS, that decouples the task into global-local semantic-guided feature separation and structure-preserving acoustic reconstruction. Our approach introduces a dual-stage mechanism for semantic separation, operating on distinct global and local semantic feature spaces. We first perform global-semantic separation through a global semantic feature space aligned with text queries. A Q-Audio architecture is employed to align audio and text modalities, serving as pretrained global-semantic encoders. Conditioned on the predicted global feature, we then perform the second-stage local-semantic separation on AudioMAE features that preserve time-frequency structures, followed by acoustic reconstruction. We also propose an instruction processing pipeline to parse arbitrary text queries into structured operations, extraction or removal, coupled with audio descriptions, enabling flexible sound manipulation. Our method achieves state-of-the-art separation performance with data-efficient training while maintaining superior semantic consistency with queries in complex auditory scenes.

Figures (8)

• Figure 1: Comparison between previous blind single-stage and our hierarchical modeling frameworks.
• Figure 2: Overview of our proposed hierarchical modeling and separation frameworks. (a) Two-level audio feature representation. (b) Text-queried two-level separation and decoupled semantic separation and acoustic reconstruction.
• Figure 3: Overview of the semantic-to-acoustic and acoustic decoder. The autoregressive transformer generates acoustic tokens by our neural codec TF-Codec, conditioned on local semantic features.
• Figure 4: The pipeline of processing arbitrary text instructions.
• Figure 5: The trends of KL and AFSim with varying source overlap ratios.