Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enroll-on-Wakeup: A First Comparative Study of Target Speech Extraction for Seamless Interaction in Real Noisy Human-Machine Dialogue Scenarios

Yiming Yang, Guangyong Wang, Haixin Guan, Yanhua Long

TL;DR

This paper proposes Enroll-on-Wakeup (EoW), a novel framework where the wake-word segment, captured naturally during human-machine interaction, is automatically utilized as the enrollment reference, which eliminates the need for pre-collected speech to enable a seamless experience.

Abstract

Target speech extraction (TSE) typically relies on pre-recorded high-quality enrollment speech, which disrupts user experience and limits feasibility in spontaneous interaction. In this paper, we propose Enroll-on-Wakeup (EoW), a novel framework where the wake-word segment, captured naturally during human-machine interaction, is automatically utilized as the enrollment reference. This eliminates the need for pre-collected speech to enable a seamless experience. We perform the first systematic study of EoW-TSE, evaluating advanced discriminative and generative models under real diverse acoustic conditions. Given the short and noisy nature of wake-word segments, we investigate enrollment augmentation using LLM-based TTS. Results show that while current TSE models face performance degradation in EoW-TSE, TTS-based assistance significantly enhances the listening experience, though gaps remain in speech recognition accuracy.

Enroll-on-Wakeup: A First Comparative Study of Target Speech Extraction for Seamless Interaction in Real Noisy Human-Machine Dialogue Scenarios

TL;DR

This paper proposes Enroll-on-Wakeup (EoW), a novel framework where the wake-word segment, captured naturally during human-machine interaction, is automatically utilized as the enrollment reference, which eliminates the need for pre-collected speech to enable a seamless experience.

Abstract

Target speech extraction (TSE) typically relies on pre-recorded high-quality enrollment speech, which disrupts user experience and limits feasibility in spontaneous interaction. In this paper, we propose Enroll-on-Wakeup (EoW), a novel framework where the wake-word segment, captured naturally during human-machine interaction, is automatically utilized as the enrollment reference. This eliminates the need for pre-collected speech to enable a seamless experience. We perform the first systematic study of EoW-TSE, evaluating advanced discriminative and generative models under real diverse acoustic conditions. Given the short and noisy nature of wake-word segments, we investigate enrollment augmentation using LLM-based TTS. Results show that while current TSE models face performance degradation in EoW-TSE, TTS-based assistance significantly enhances the listening experience, though gaps remain in speech recognition accuracy.
Paper Structure (16 sections, 2 equations, 3 figures, 4 tables)

This paper contains 16 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Conventional TSE Formulation
  4. The EoW-TSE Architecture and Challenges
  5. Enroll-on-Wakeup TSE Systems
  6. Scenario Description
  7. Target Speech Extraction Systems
  8. Discriminative TSE Models
  9. Generative TSE Model
  10. Augment Enrollment with LLM-based TTS Models
  11. Experimental Setup
  12. Results and Discussion
  13. Results on Libri2mix 2spk+noise Condition
  14. Overall Performance of EoW-TSE Systems
  15. Effect of Synthetic Enrollment
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Illustration of EoW-TSE system.
  • Figure 2: OVRL scores on five scenarios: Original EoW-TSE vs. TTS-augmented (CR).
  • Figure 3: WERs on five scenarios: Original EoW-TSE vs. TTS-augmented (CR).