An Investigation on Speaker Augmentation for End-to-End Speaker Extraction

TL;DR

This work tackles target confusion in end-to-end speaker extraction by introducing a simple yet effective speaker augmentation pipeline that generates pseudo-speakers through time-domain resampling and rescaling. The augmentation preserves content, tempo, and prosody while altering speaker traits, creating hard samples that push the model to learn genuine speaker characteristics. Across WSJ0-2Mix and LibriMix, and for two state-of-the-art architectures, the method consistently improves SI-SDRi and reduces NSR, with larger gains in challenging conditions. Moreover, the augmentation is compatible with metric learning, offering additional performance gains and highlighting the practical value of expanding speaker diversity to boost E2E-SE robustness.

Abstract

Target confusion, defined as occasional switching to non-target speakers, poses a key challenge for end-to-end speaker extraction (E2E-SE) systems. We argue that this problem is largely caused by the lack of generalizability and discrimination of the speaker embeddings, and introduce a simple yet effective speaker augmentation strategy to tackle the problem. Specifically, we propose a time-domain resampling and rescaling pipeline that alters speaker traits while preserving other speech properties. This generates a variety of pseudo-speakers to help establish a generalizable speaker embedding space, while the speaker-trait-specific augmentation creates hard samples that force the model to focus on genuine speaker characteristics. Experiments on WSJ0-2Mix and LibriMix show that our method mitigates the target confusion and improves extraction performance. Moreover, it can be combined with metric learning, another effective approach to address target confusion, leading to further gains.