Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlowSep: Language-Queried Sound Separation with Rectified Flow Matching

Yi Yuan, Xubo Liu, Haohe Liu, Mark D. Plumbley, Wenwu Wang

TL;DR

FlowSep is introduced, a new generative model based on RFM for LASS tasks that outperforms the state-of-the-art models across multiple benchmarks, as evaluated with subjective and objective metrics.

Abstract

Language-queried audio source separation (LASS) focuses on separating sounds using textual descriptions of the desired sources. Current methods mainly use discriminative approaches, such as time-frequency masking, to separate target sounds and minimize interference from other sources. However, these models face challenges when separating overlapping soundtracks, which may lead to artifacts such as spectral holes or incomplete separation. Rectified flow matching (RFM), a generative model that establishes linear relations between the distribution of data and noise, offers superior theoretical properties and simplicity, but has not yet been explored in sound separation. In this work, we introduce FlowSep, a new generative model based on RFM for LASS tasks. FlowSep learns linear flow trajectories from noise to target source features within the variational autoencoder (VAE) latent space. During inference, the RFM-generated latent features are reconstructed into a mel-spectrogram via the pre-trained VAE decoder, followed by a pre-trained vocoder to synthesize the waveform. Trained on 1,680 hours of audio data, FlowSep outperforms the state-of-the-art models across multiple benchmarks, as evaluated with subjective and objective metrics. Additionally, our results show that FlowSep surpasses a diffusion-based LASS model in both separation quality and inference efficiency, highlighting its strong potential for audio source separation tasks. Code, pre-trained models and demos can be found at: https://audio-agi.github.io/FlowSep_demo/ .

FlowSep: Language-Queried Sound Separation with Rectified Flow Matching

TL;DR

FlowSep is introduced, a new generative model based on RFM for LASS tasks that outperforms the state-of-the-art models across multiple benchmarks, as evaluated with subjective and objective metrics.

Abstract

Language-queried audio source separation (LASS) focuses on separating sounds using textual descriptions of the desired sources. Current methods mainly use discriminative approaches, such as time-frequency masking, to separate target sounds and minimize interference from other sources. However, these models face challenges when separating overlapping soundtracks, which may lead to artifacts such as spectral holes or incomplete separation. Rectified flow matching (RFM), a generative model that establishes linear relations between the distribution of data and noise, offers superior theoretical properties and simplicity, but has not yet been explored in sound separation. In this work, we introduce FlowSep, a new generative model based on RFM for LASS tasks. FlowSep learns linear flow trajectories from noise to target source features within the variational autoencoder (VAE) latent space. During inference, the RFM-generated latent features are reconstructed into a mel-spectrogram via the pre-trained VAE decoder, followed by a pre-trained vocoder to synthesize the waveform. Trained on 1,680 hours of audio data, FlowSep outperforms the state-of-the-art models across multiple benchmarks, as evaluated with subjective and objective metrics. Additionally, our results show that FlowSep surpasses a diffusion-based LASS model in both separation quality and inference efficiency, highlighting its strong potential for audio source separation tasks. Code, pre-trained models and demos can be found at: https://audio-agi.github.io/FlowSep_demo/ .
Paper Structure (20 sections, 3 equations, 3 figures, 3 tables)

This paper contains 20 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Text Encoder
  4. Latent Feature Generator
  5. Rectified Flow Matching
  6. Channel-Conditioned Generation
  7. VAE Decoder and GAN Vocoder
  8. Dataset and Evaluation metrics
  9. Dataset
  10. Training Set
  11. Test Set
  12. Evaluation Metrics
  13. Experiments and Results
  14. Data Processing
  15. Experimental Details
  16. ...and 5 more sections

Figures (3)

  • Figure 1: The architecture of FlowSep. FlowSep consists of four main components: (1) a FLAN-T5 encoder for text embedding; (2) a VAE for encoding and decoding mel-spectrograms; (3) an RFM module for generating audio features within the VAE latent space; (4) a BigVGAN vocoder to generate the waveform.
  • Figure 2: The channel-concatenation conditioning mechanism
  • Figure 3: A case study of separation results on DE-S test set, as compared with the ground truth. More results can be found online from https://audio-agi.github.io/FlowSep_demo/.