DualStream Contextual Fusion Network: Efficient Target Speaker Extraction by Leveraging Mixture and Enrollment Interactions

TL;DR

The paper tackles target speaker extraction under multi-party conditions by leveraging contextualized enrollment beyond fixed speaker embeddings. It introduces DCF-Net, a time-frequency domain network featuring a DualStream Fusion Block (DSFB) that jointly processes mixture and enrollment representations using an MGI mechanism and SE-based channel recalibration. A dual-path transformer-based extraction network estimates a target mask, with a decoder reconstructing the target speech; training minimizes SI-SDR-based loss. Empirical results on WSJ0-2Mix, WHAM!, and WHAMR! show state-of-the-art performance and robustness to noise and reverberation, with a notably reduced target confusion rate of 0.4%. The work demonstrates practical potential for efficient, context-aware target extraction in real-world acoustic scenes.

Abstract

Target speaker extraction focuses on extracting a target speech signal from an environment with multiple speakers by leveraging an enrollment. Existing methods predominantly rely on speaker embeddings obtained from the enrollment, potentially disregarding the contextual information and the internal interactions between the mixture and enrollment. In this paper, we propose a novel DualStream Contextual Fusion Network (DCF-Net) in the time-frequency (T-F) domain. Specifically, DualStream Fusion Block (DSFB) is introduced to obtain contextual information and capture the interactions between contextualized enrollment and mixture representation across both spatial and channel dimensions, and then rich and consistent representations are utilized to guide the extraction network for better extraction. Experimental results demonstrate that DCF-Net outperforms state-of-the-art (SOTA) methods, achieving a scale-invariant signal-to-distortion ratio improvement (SI-SDRi) of 21.6 dB on the benchmark dataset, and exhibits its robustness and effectiveness in both noise and reverberation scenarios. In addition, the wrong extraction results of our model, called target confusion problem, reduce to 0.4%, which highlights the potential of DCF-Net for practical applications.

Figures (5)

• Figure 1: TSE network leveraging speaker embeddings.
• Figure 2: The overall architecture of our proposed network. There are four main components, different capital letters represent different parts of our model.
• Figure 3: Multi-range 2D convolution.
• Figure 4: The target confusion problem (TCP) in CIENet $(a)$ and DCF-Net $(b)$. These two are the distribution in terms of SI-SDRi on WSJ0-2Mix test set, each axis corresponds to a speaker. $s1$, $s2$$<$$0dB$ denotes that SI-SDRi of both $s1$ and $s2$ are below $0dB$, while $s1/s2 < 0dB$ means that either of them is below $0dB$, both of two mean the target confusion problem happens.
• Figure 5: Performance comparison of varying numbers of DSFB (O) on the WSJ0-2Mix dataset, including parameter(M) count.