Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

Haoyang Li, Xuyi Zhuang, Azmat Adnan, Ye Ni, Wei Rao, Shreyas Gopal, Eng Siong Chng

TL;DR

This work tackles Target Speaker Extraction under distribution shifts by introducing GenTSE, a fully generative two-stage decoder-only LM framework. It splits the task into a semantic stage that predicts coarse $S$ tokens from continuous embeddings and an acoustic stage that generates fine $A$ tokens conditioned on $S$, using a single-codebook SimCodec for reconstruction. To address exposure bias and perceptual misalignment, the authors introduce Frozen-LM Conditioning (FLC) and Direct Preference Optimization (DPO), respectively, achieving more natural and speaker-consistent outputs. Experiments on Libri2Mix show GenTSE outperforming previous LM-based methods in quality, intelligibility, and speaker fidelity, with ablations confirming the value of the semantic stage and the chosen input representations. The approach offers a scalable, high-fidelity path toward robust, perceptually aligned TSE in real-world, multi-speaker scenarios.

Abstract

Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech. We present GenTSE, a two-stage decoder-only generative LM approach for TSE: Stage-1 predicts coarse semantic tokens, and Stage-2 generates fine acoustic tokens. Separating semantics and acoustics stabilizes decoding and yields more faithful, content-aligned target speech. Both stages use continuous SSL or codec embeddings, offering richer context than discretized-prompt methods. To reduce exposure bias, we employ a Frozen-LM Conditioning training strategy that conditions the LMs on predicted tokens from earlier checkpoints to reduce the gap between teacher-forcing training and autoregressive inference. We further employ DPO to better align outputs with human perceptual preferences. Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.

GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

TL;DR

This work tackles Target Speaker Extraction under distribution shifts by introducing GenTSE, a fully generative two-stage decoder-only LM framework. It splits the task into a semantic stage that predicts coarse tokens from continuous embeddings and an acoustic stage that generates fine tokens conditioned on , using a single-codebook SimCodec for reconstruction. To address exposure bias and perceptual misalignment, the authors introduce Frozen-LM Conditioning (FLC) and Direct Preference Optimization (DPO), respectively, achieving more natural and speaker-consistent outputs. Experiments on Libri2Mix show GenTSE outperforming previous LM-based methods in quality, intelligibility, and speaker fidelity, with ablations confirming the value of the semantic stage and the chosen input representations. The approach offers a scalable, high-fidelity path toward robust, perceptually aligned TSE in real-world, multi-speaker scenarios.

Abstract

Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech. We present GenTSE, a two-stage decoder-only generative LM approach for TSE: Stage-1 predicts coarse semantic tokens, and Stage-2 generates fine acoustic tokens. Separating semantics and acoustics stabilizes decoding and yields more faithful, content-aligned target speech. Both stages use continuous SSL or codec embeddings, offering richer context than discretized-prompt methods. To reduce exposure bias, we employ a Frozen-LM Conditioning training strategy that conditions the LMs on predicted tokens from earlier checkpoints to reduce the gap between teacher-forcing training and autoregressive inference. We further employ DPO to better align outputs with human perceptual preferences. Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.
Paper Structure (14 sections, 5 equations, 2 figures, 3 tables)

This paper contains 14 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Hierarchical Modeling
  4. Semantic extraction stage
  5. Acoustic generation stage
  6. Frozen-LM Conditioning
  7. Direct Preference Optimization
  8. Experiments
  9. Dataset
  10. Evaluation Metrics
  11. Implementation Details
  12. Baseline Models
  13. Results and discussion
  14. Conclusion

Figures (2)

  • Figure 1: The overall architecture of GenTSE consisting of a semantic extraction stage (left) and an acoustic generation stage (right)
  • Figure 2: Semantic LM validation top-1 accuracy during FLC: blue = teacher-forcing, green = autoregressive.