Unified Architecture and Unsupervised Speech Disentanglement for Speaker Embedding-Free Enrollment in Personalized Speech Enhancement

TL;DR

This paper tackles the dual challenges of conventional speech enhancement (SE) and personalized speech enhancement (PSE) by proposing a unified, lightweight framework that eliminates the need for external speaker embeddings. It introduces USEF-PNet, a single architecture capable of performing SE and PSE via mixed training batches, and DSEF-PNet, which employs unsupervised enrollment speech disentanglement through HEIT to stabilize target speech extraction across varying enrollment conditions; it also investigates Long-Short Enrollment Pairing (LSEP) to address short enrollment scenarios. Empirical results on Libri2Mix and VoiceBank-DEMAND demonstrate that the unified model matches or surpasses task-specific baselines, with UDSEF-PNet achieving notable gains in PESQ and STOI while maintaining SISDR, and DSEF-PNet delivering robustness to enrollment variability. The study shows that random enrollment durations during training generally offer better generalization than fixed-duration strategies, and confirms that disentanglement-based training can substantially improve PSE robustness without adding inference overhead, paving the way for practical, universal SE/PSE deployments.

Abstract

Conventional speech enhancement (SE) aims to improve speech perception and intelligibility by suppressing noise without requiring enrollment speech as reference, whereas personalized SE (PSE) addresses the cocktail party problem by extracting a target speaker's speech using enrollment speech. While these two tasks tackle different yet complementary challenges in speech signal processing, they often share similar model architectures, with PSE incorporating an additional branch to process enrollment speech. This suggests developing a unified model capable of efficiently handling both SE and PSE tasks, thereby simplifying deployment while maintaining high performance. However, PSE performance is sensitive to variations in enrollment speech, like emotional tone, which limits robustness in real-world applications. To address these challenges, we propose two novel models, USEF-PNet and DSEF-PNet, both extending our previous SEF-PNet framework. USEF-PNet introduces a unified architecture for processing enrollment speech, integrating SE and PSE into a single framework to enhance performance and streamline deployment. Meanwhile, DSEF-PNet incorporates an unsupervised speech disentanglement approach by pairing a mixture speech with two different enrollment utterances and enforcing consistency in the extracted target speech. This strategy effectively isolates high-quality speaker identity information from enrollment speech, reducing interference from factors such as emotion and content, thereby improving PSE robustness. Additionally, we explore a long-short enrollment pairing (LSEP) strategy to examine the impact of enrollment speech duration during both training and evaluation. Extensive experiments on the Libri2Mix and VoiceBank DEMAND demonstrate that our proposed USEF-PNet, DSEF-PNet all achieve substantial performance improvements, with random enrollment duration performing slightly better.

Figures (3)

• Figure 1: Overview of our previously proposed SEF-PNet model. All colored blocks highlight the key contributions over the original sDPCCN.
• Figure 2: Structure of Unified Architecture on SEF-PNet (USEF-PNet).
• Figure 3: The whole training framework of DSEF-PNet with heterogeneous enrollment invariant training (HEIT).