Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR

TL;DR

This work addresses target speech extraction (TSE) by overcoming limitations of mask-based discriminative methods and intelligibility gaps in generative approaches. It introduces a Whisper-based generative TSE framework with a shared target speech encoder prompted by target speaker cues and enrollment speech, a flow-based token-to-spectrogram synthesizer via optimal-transport conditional flow matching, and a text decoder trained to predict transcripts for enhanced intelligibility. The model is trained with a joint objective combining flow-based synthesis and transcript learning, while fine-tuning only selected components through LoRA for efficiency. Experiments on Libri2Mix and WSJ0-2mix demonstrate improved perceptual quality and intelligibility over baselines, with ablations confirming the importance of enrollment cues and joint optimization; however, some non-linguistic detail distortions and higher inference costs remain, motivating future lightweight and real-time strategies.

Abstract

Target speech extraction (TSE) isolates the speech of a specific speaker from a multi-talker overlapped speech mixture. Most existing TSE models rely on discriminative methods, typically predicting a time-frequency spectrogram mask for the target speech. However, imperfections in these masks often result in over-/under-suppression of target/non-target speech, degrading perceptual quality. Generative methods, by contrast, re-synthesize target speech based on the mixture and target speaker cues, achieving superior perceptual quality. Nevertheless, these methods often overlook speech intelligibility, leading to alterations or loss of semantic content in the re-synthesized speech. Inspired by the Whisper model's success in target speaker ASR, we propose a generative TSE framework based on the pre-trained Whisper model to address the above issues. This framework integrates semantic modeling with flow-based acoustic modeling to achieve both high intelligibility and perceptual quality. Results from multiple benchmarks demonstrate that the proposed method outperforms existing generative and discriminative baselines. We present speech samples on https://aisaka0v0.github.io/GenerativeTSE_demo/.

Figures (3)

• Figure 1: Overview of the proposed method. The solid line represents the forward propagation flow; the dashed line represents the backpropagation flow.
• Figure 2: Normalized metrics versus LoRA rank. All metrics are evaluated with WhisperTSE-S on Libri2Mix-dev-clean.
• Figure 3: Spectrogram Visualization