Speech Separation with Pretrained Frontend to Minimize Domain Mismatch

TL;DR

A self-supervised domain-invariant pretrained (DIP) frontend that is exposed to mixture data without the need for target reference speech and introduces a novel separation pipeline to align the feature resolution of the separation models.

Abstract

Speech separation seeks to separate individual speech signals from a speech mixture. Typically, most separation models are trained on synthetic data due to the unavailability of target reference in real-world cocktail party scenarios. As a result, there exists a domain gap between real and synthetic data when deploying speech separation models in real-world applications. In this paper, we propose a self-supervised domain-invariant pretrained (DIP) frontend that is exposed to mixture data without the need for target reference speech. The DIP frontend utilizes a Siamese network with two innovative pretext tasks, mixture predictive coding (MPC) and mixture invariant coding (MIC), to capture shared contextual cues between real and synthetic unlabeled mixtures. Subsequently, we freeze the DIP frontend as a feature extractor when training the downstream speech separation models on synthetic data. By pretraining the DIP frontend with the contextual cues, we expect that the speech separation skills learned from synthetic data can be effectively transferred to real data. To benefit from the DIP frontend, we introduce a novel separation pipeline to align the feature resolution of the separation models. We evaluate the speech separation quality on standard benchmarks and real-world datasets. The results confirm the superiority of our DIP frontend over existing speech separation models. This study underscores the potential of large-scale pretraining to enhance the quality and intelligibility of speech separation in real-world applications.

Figures (5)

• Figure 1: The architecture of our DIP frontend. We pretrain the frontend with both real and synthetic mixtures through a Siamese network to capture the contextual cues. The $\boldsymbol{x}$ and $\boldsymbol{y}$ are randomly sampled mixture waveforms from the real mixtures and synthetic datasets. The $\mathbf{E}_x$ and $\mathbf{E}_y$ are the local embedding extracted by the local encoder $h_{\theta}$. The $\tilde{E}_x$ and $\tilde{E}_y$ are the randomly masked embedding with percentage $\delta$. The $\mathbf{Z}_x$ and $\mathbf{Z}_y$ are the quantized embeding with codebook $q_{\theta}$ for siamses network to predict. The $\mathbf{C}_x$ and $\mathbf{C}_y$ are contextual cues captured by the Transformer $g_{\theta}$. The $\mathcal{L}_{MPC}$ is the contrastive loss defined at Eq. \ref{['equ.mpc']} and $\mathcal{L}_{MMD}$ is the discrepancy loss in Eq. \ref{['equ.mmd_rep']}.
• Figure 2: A universal separation training pipeline that incorporates various frontends with diverse speech separation models. A frontend is employed to learn the contextual information, that is followed by an adaptation layer that upsamples the input sequence to time-align with the encoder output for the downstream separation models. The frontend is pre-trained and frozen in the training pipeline. Here $y$ is the two-speaker synthetic mixture waveform. $V_y$ and $\tilde{V_y}$ are the auditory features and contextual cues of mixture waveform. $\hat{M_1}$ and $\hat{M_2}$ are predicted masks for two speakers, respectively, and $\hat{s_1}$, $\hat{s_2}$ are two reconstructed target reference speech.
• Figure 3: The SI-SDRi(dB) of different methods to separate mixture without from the Vox2Mix test set access to the clean reference speech. The "Baseline" is the ConvTasNet without any frontend, the "DIP" is the ConvTasNet with our DIP frontend, and the "Supervised" is the ConvTasNet trained on Vox2Mix dataset without any frontend.
• Figure 4: The SI-SDRi(dB) score of the scores on individual utterances. The red bar is the number of samples for special SI-SDRi(dB) using the ConvTasNet baseline model. The blue bar on the ConvTasNet with our DIP frontend.
• Figure 5: The SI-SDRi(dB) of three different time-domain speech separators to transfer knowledge from LM2Mix to Vox2Mix dataset. The blue bar is the separator without any pretrained frontend and the red bar is the separator with our DIP frontend.