Discriminative-Generative Target Speaker Extraction with Decoder-Only Language Models

TL;DR

LauraTSE is proposed, a generative TSE model built upon an auto-regressive decoder-only language model that effectively combines the robustness and controllability of discriminative models with the superior naturalness and quality enhancement capabilities of generative models.

Abstract

Target speaker extraction (TSE) aims to recover the speech signal of a desired speaker from a mixed audio recording, given a short enrollment utterance. Most existing TSE approaches are based on discriminative modeling paradigms. Although effective at suppressing interfering speakers, these methods often struggle to produce speech with high perceptual quality and naturalness. To address this limitation, we first propose LauraTSE, a generative TSE model built upon an auto-regressive decoder-only language model. However, purely generative approaches may suffer from hallucinations, content drift, and limited controllability, which may undermine their reliability in complex acoustic scenarios. To overcome these challenges, we further introduce a discriminative-generative TSE framework. In this framework, a discriminative front-end is employed to robustly extract the target speaker's speech, yielding stable and controllable intermediate representations. A generative back-end then operates in the neural audio codec representation space to reconstruct fine-grained speech details and enhance perceptual quality. This two-stage design effectively combines the robustness and controllability of discriminative models with the superior naturalness and quality enhancement capabilities of generative models. Moreover, we systematically investigate collaborative training strategies for the proposed framework, including freezing or fine-tuning the front-end, incorporating an auxiliary SI-SDR loss, and exploring both auto-regressive and non-auto-regressive inference mechanisms. Experimental results demonstrate that the proposed framework achieves a more favorable trade-off among speech quality, intelligibility, and speaker consistency.

Figures (6)

• Figure 1: The diagram of a typical target speaker extraction method. The speaker embedding extractor is typically a pre-trained speaker recognition model. 'C' denotes the concatenation.
• Figure 2: The diagram of LauraTSE network. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively. We use two weight sharing conformer to process the mixed and reference speech separately.
• Figure 3: The diagram of discriminative-generative target speaker extraction framework. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively.
• Figure 4: The diagram of USEF-Laura-TSE. ‘m’ and ‘r’ denote the mixed speech and reference speech, respectively.
• Figure 5: dWER versus training data scale across models. Annotations "(-X)" denote relative dWER reduction (percentage points) compared to the preceding smaller dataset.