Training-Free Multi-Step Inference for Target Speaker Extraction

TL;DR

Joint metric optimization is introduced to balance the objectives of target speaker extraction and controllable extraction preferences for practical deployment, enabling controllable extraction preferences for practical deployment.

Abstract

Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.

Figures (1)

• Figure 1: Overall architecture of the proposed multi-step inference TSE. The frozen model $f_\theta$ is shared across iterations to progressively refine the target estimate via input interpolation and non-intrusive joint selection.