Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Duojia Li, Shuhan Zhang, Zihan Qian, Wenxuan Wu, Shuai Wang, Qingyang Hong, Lin Li, Haizhou Li

TL;DR

AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective, improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).

Abstract

In target speaker extraction (TSE), we aim to recover target speech from a multi-talker mixture using a short enrollment utterance as reference. Recent studies on diffusion and flow-matching generators have improved target-speech fidelity. However, multi-step sampling increases latency, and one-step solutions often rely on a mixture-dependent time coordinate that can be unreliable for real-world conversations. We present AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective. AlphaFlowTSE learns mean-velocity transport along a mixture-to-target trajectory starting from the observed mixture, eliminating auxiliary mixing-ratio prediction, and stabilizes training by combining flow matching with an interval-consistency teacher-student target. Experiments on Libri2Mix and REAL-T confirm that AlphaFlowTSE improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).

AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

TL;DR

AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective, improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).

Abstract

In target speaker extraction (TSE), we aim to recover target speech from a multi-talker mixture using a short enrollment utterance as reference. Recent studies on diffusion and flow-matching generators have improved target-speech fidelity. However, multi-step sampling increases latency, and one-step solutions often rely on a mixture-dependent time coordinate that can be unreliable for real-world conversations. We present AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective. AlphaFlowTSE learns mean-velocity transport along a mixture-to-target trajectory starting from the observed mixture, eliminating auxiliary mixing-ratio prediction, and stabilizes training by combining flow matching with an interval-consistency teacher-student target. Experiments on Libri2Mix and REAL-T confirm that AlphaFlowTSE improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).
Paper Structure (19 sections, 24 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 24 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Flow-based Generative TSE
  4. One-Step Generative TSE
  5. AlphaFlow: JVP-Free Interval Consistency for Mean-Velocity Models
  6. Method
  7. AlphaFlowTSE
  8. JVP-Free AlphaFlow Training
  9. Inference
  10. Experiment
  11. Datasets and Data Preparation
  12. Experimental Setup
  13. Evaluation Metrics
  14. Results
  15. Libri2Mix: One-Step Benchmark Performance
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Overall architecture of AlphaFlowTSE. Given a mixture waveform $y$ and an enrollment utterance $e$, we compute complex STFT features and form the mixture feature $Y$ and enrollment feature $E$ (real/imaginary concatenation). During training, the backbone takes the current state feature $z_t$; during inference we initialize $z_0=Y$. The enrollment feature is concatenated as a temporal prefix, yielding $[E\!\parallel\! z_t]$ (or $[E\!\parallel\! z_0]$ at inference), which is fed to the UDiT backbone. The backbone is conditioned via AdaLN on the absolute time $t$ and the interval length $\Delta=r-t$ (with $r=1$ at inference), and predicts the mean velocity for finite-interval transport, denoted $u_\theta(t,r,[E\!\parallel\! z_t])$. One-step inference (NFE$=1$) produces an estimated complex STFT $\hat{S}=(\hat{S}_{\mathrm{Re}},\hat{S}_{\mathrm{Im}})$, which is converted to the target waveform $\hat{s}$ by iSTFT. The dashed module is an optional mixing-ratio predictor used only in the background-to-target ablation to predict the start coordinate $\hat{\tau}$.
  • Figure 2: ASR error rates on REAL-T under two inference settings: (a) w/o MR predictor and (b) w/ MR predictor. Left panels report English WER (average and subsets: AMI, CHiME-6, DipCo), and right panels report Chinese CER (average and subsets: AISHELL-4, AliMeeting) for AD-FlowTSE, MeanFlowTSE, and AlphaFlowTSE. Lower is better.
  • Figure 3: Speaker similarity (SIM / SpkSim) on REAL-T under two inference settings: (a) MR-free and (b) w/ MR predictor. Left panels report English SpkSim (average and subsets: AMI, CHiME-6, DipCo), and right panels report Chinese SpkSim (average and subsets: AISHELL-4, AliMeeting) for AD-FlowTSE, MeanFlowTSE, and AlphaFlowTSE. Higher is better.