Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LuSeeL: Language-queried Binaural Universal Sound Event Extraction and Localization

Zexu Pan, Shengkui Zhao, Yukun Ma, Haoxu Wang, Yiheng Jiang, Biao Tian, Bin Ma

TL;DR

LuSeeL introduces a language-queried binaural framework that jointly extracts and localizes universal sound events from binaural mixtures. By coupling a dual-domain transformer-based extractor with a text-conditioned modulation and a DoA-focused localization pathway, the model leverages spatial cues to improve extraction accuracy and DoA estimation. End-to-end training with a hybrid time-frequency loss and DoA supervision yields superior performance over single-task baselines and ablations, across 2- and 3-source AudioCaps mixtures. This approach demonstrates the value of integrating text prompts, spatial information, and joint optimization for robust auditory scene understanding in realistic 3D environments.

Abstract

Most universal sound extraction algorithms focus on isolating a target sound event from single-channel audio mixtures. However, the real world is three-dimensional, and binaural audio, which mimics human hearing, can capture richer spatial information, including sound source location. This spatial context is crucial for understanding and modeling complex auditory scenes, as it inherently informs sound detection and extraction. In this work, we propose a language-driven universal sound extraction network that isolates text-described sound events from binaural mixtures by effectively leveraging the spatial cues present in binaural signals. Additionally, we jointly predict the direction of arrival (DoA) of the target sound using spatial features from the extraction network. This dual-task approach exploits complementary location information to improve extraction performance while enabling accurate DoA estimation. Experimental results on the in-the-wild AudioCaps dataset show that our proposed LuSeeL model significantly outperforms single-channel and uni-task baselines.

LuSeeL: Language-queried Binaural Universal Sound Event Extraction and Localization

TL;DR

LuSeeL introduces a language-queried binaural framework that jointly extracts and localizes universal sound events from binaural mixtures. By coupling a dual-domain transformer-based extractor with a text-conditioned modulation and a DoA-focused localization pathway, the model leverages spatial cues to improve extraction accuracy and DoA estimation. End-to-end training with a hybrid time-frequency loss and DoA supervision yields superior performance over single-task baselines and ablations, across 2- and 3-source AudioCaps mixtures. This approach demonstrates the value of integrating text prompts, spatial information, and joint optimization for robust auditory scene understanding in realistic 3D environments.

Abstract

Most universal sound extraction algorithms focus on isolating a target sound event from single-channel audio mixtures. However, the real world is three-dimensional, and binaural audio, which mimics human hearing, can capture richer spatial information, including sound source location. This spatial context is crucial for understanding and modeling complex auditory scenes, as it inherently informs sound detection and extraction. In this work, we propose a language-driven universal sound extraction network that isolates text-described sound events from binaural mixtures by effectively leveraging the spatial cues present in binaural signals. Additionally, we jointly predict the direction of arrival (DoA) of the target sound using spatial features from the extraction network. This dual-task approach exploits complementary location information to improve extraction performance while enabling accurate DoA estimation. Experimental results on the in-the-wild AudioCaps dataset show that our proposed LuSeeL model significantly outperforms single-channel and uni-task baselines.
Paper Structure (17 sections, 4 equations, 4 figures, 1 table)

This paper contains 17 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. LuSeeL
  3. Model architecture
  4. Text encoder module
  5. Audio encoder and decoder modules
  6. Signal extractor module
  7. Localization module
  8. Objective functions
  9. Experimental setup
  10. Dataset
  11. Baselines
  12. Network configurations
  13. Optimization
  14. Results
  15. Comparison with baselines
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Our proposed LuSeeL, which jointly performs binaural Universal Sound Event Extraction and Localization using language descriptions. The symbols $\otimes$ and $\ominus$ represent element-wise multiplication and channel-wise concatenation, respectively.
  • Figure 2: The 2-source SI-SNRi histogram of various inter-source separation angles.
  • Figure 3: The 2-source MAE histogram of various inter-source separation angles.
  • Figure 4: The 2-source MAE against SI-SNRi scatter plot.