Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction

Xiang Hao, Jibin Wu, Jianwei Yu, Chenglin Xu, Kay Chen Tan

TL;DR

This work represents the first integration of LLMs with TSE, potentially establishing a new benchmark in solving the cocktail party problem and expanding the scope of TSE applications by providing a versatile, privacy-conscious solution.

Abstract

Humans can easily isolate a single speaker from a complex acoustic environment, a capability referred to as the "Cocktail Party Effect." However, replicating this ability has been a significant challenge in the field of target speaker extraction (TSE). Traditional TSE approaches predominantly rely on voiceprints, which raise privacy concerns and face issues related to the quality and availability of enrollment samples, as well as intra-speaker variability. To address these issues, this work introduces a novel text-guided TSE paradigm named LLM-TSE. In this paradigm, a state-of-the-art large language model, LLaMA 2, processes typed text input from users to extract semantic cues. We demonstrate that textual descriptions alone can effectively serve as cues for extraction, thus addressing privacy concerns and reducing dependency on voiceprints. Furthermore, our approach offers flexibility by allowing the user to specify the extraction or suppression of a speaker and enhances robustness against intra-speaker variability by incorporating context-dependent textual information. Experimental results show competitive performance with text-based cues alone and demonstrate the effectiveness of using text as a task selector. Additionally, they achieve a new state-of-the-art when combining text-based cues with pre-registered cues. This work represents the first integration of LLMs with TSE, potentially establishing a new benchmark in solving the cocktail party problem and expanding the scope of TSE applications by providing a versatile, privacy-conscious solution.

Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction

TL;DR

This work represents the first integration of LLMs with TSE, potentially establishing a new benchmark in solving the cocktail party problem and expanding the scope of TSE applications by providing a versatile, privacy-conscious solution.

Abstract

Humans can easily isolate a single speaker from a complex acoustic environment, a capability referred to as the "Cocktail Party Effect." However, replicating this ability has been a significant challenge in the field of target speaker extraction (TSE). Traditional TSE approaches predominantly rely on voiceprints, which raise privacy concerns and face issues related to the quality and availability of enrollment samples, as well as intra-speaker variability. To address these issues, this work introduces a novel text-guided TSE paradigm named LLM-TSE. In this paradigm, a state-of-the-art large language model, LLaMA 2, processes typed text input from users to extract semantic cues. We demonstrate that textual descriptions alone can effectively serve as cues for extraction, thus addressing privacy concerns and reducing dependency on voiceprints. Furthermore, our approach offers flexibility by allowing the user to specify the extraction or suppression of a speaker and enhances robustness against intra-speaker variability by incorporating context-dependent textual information. Experimental results show competitive performance with text-based cues alone and demonstrate the effectiveness of using text as a task selector. Additionally, they achieve a new state-of-the-art when combining text-based cues with pre-registered cues. This work represents the first integration of LLMs with TSE, potentially establishing a new benchmark in solving the cocktail party problem and expanding the scope of TSE applications by providing a versatile, privacy-conscious solution.
Paper Structure (34 sections, 4 equations, 4 figures, 1 table)

This paper contains 34 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Speech Separation and Target Speaker Extraction
  4. Audio-Language Multimodal Modeling
  5. Audio-Language-Vison Multimodal Target Source Separation
  6. Application Scenarios Enabled by LLM-TSE
  7. Use Text as Transcription Snippets
  8. Use Text as Semantic Description
  9. Use Text as a Task Selector
  10. Use Text to Complement Pre-registered Cues
  11. LLM-TSE Model
  12. Mixture Encoder and Decoder
  13. Text Cue Encoder
  14. Audio Cue Encoder
  15. Fusion Layer
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Comparison between conventional TSE system and our proposed Text-Guided TSE system. The former relies on the pre-registered voiceprint of the target speaker as an extraction cue, while our system offers flexibility to incorporate text-based cues to facilitate target speaker extraction.
  • Figure 2: New application scenarios enabled by the proposed LLM-TSE model. The central part is a mixture audio sample where two speakers' voices overlap. The male speaker, although positioned at a greater distance from the microphone, has a voice with higher volume and is saying "Happy Mid-Autumn Festival". In contrast, the female speaker is nearer to the microphone but speaks in a quieter tone, delivering the message "Paris 2024 Summer Olympics are scheduled to take place on July 26, 2024". The illustration's four corners show the innovative application scenarios enabled by LLM-TSE.
  • Figure 3: Overview of the proposed LLM-TSE model architecture. We use LoRA hu_lora_2021 to fine-tune a small number of parameters of the LLM component.
  • Figure 4: Samples generated from the proposed LLM-TSE model. The text box contains information about the input audio mixture. The term "w/o" indicates the absence of a certain input.