Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation

TL;DR

The paper tackles real-time speech separation where causal models lack future context and degrade in quality. It introduces a causal self-supervised pretrained (CSP) frontend trained with two pretexts, Autoregressive Hybrid Prediction (AHP) and Contextual Knowledge Distillation (CKD), to embed predictive patterns from unlabeled mixtures into the frontend representations. The CSP is integrated as a frozen feature extractor with various causal separators, yielding significant gains in SI-SDRi, SDRi, PESQ, and STOI on synthetic and real datasets and improving WER in real-world settings. This approach reduces domain mismatch in streaming scenarios and enhances the practicality of real-time speech separation by leveraging predictive patterns learned from mixtures.

Abstract

Speech separation (SS) seeks to disentangle a multi-talker speech mixture into single-talker speech streams. Although SS can be generally achieved using offline methods, such a processing paradigm is not suitable for real-time streaming applications. Causal separation models, which rely only on past and present information, offer a promising solution for real-time streaming. However, these models typically suffer from notable performance degradation due to the absence of future context. In this paper, we introduce a novel frontend that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models through predictive patterns. The pretrained frontend employs a transformer decoder network with a causal convolutional encoder as the backbone and is pretrained in a self-supervised manner with two innovative pretext tasks: autoregressive hybrid prediction and contextual knowledge distillation. These tasks enable the model to capture predictive patterns directly from mixtures in a self-supervised manner. The pretrained frontend subsequently serves as a feature extractor to generate high-quality predictive patterns. Comprehensive evaluations on synthetic and real-world datasets validated the effectiveness of the proposed pretrained frontend.

Figures (6)

• Figure 1: The details of AHP pretext task. $\mathbf{z}_{t+1}$ is the future mixture latent embedding. $\mathbf{c}_{t}$ is the current predictive pattern. $\mathbf{u}_{t+1}$ is future latent token, while $\mathbf{v}_{t}$ is the current pattern token. The $\mathcal{L}_{\text{TD}}$ and $\mathcal{L}_{\text{BU}}$ are loss functions defined at Eq. \ref{['equ.ahp_td']} and Eq. \ref{['equ.ahp_bu']}, respectively.
• Figure 2: The details of CKD pretext task. The CSP frontend is the trainable causal student model, while the DIP frontend is the frozen non-causal teacher model. The $\mathbf{c}_{t}$ is the current predictive pattern. The $\overline{\mathbf{c}}_{t}$ is the current contextual representation. The $\mathcal{L}_{\text{CKD}}$ is the distillation loss in Eq. \ref{['equ.ckd']}.
• Figure 3: The architecture of CSP frontend. The causal Conv1D is the causal 1-dimensional convolutional layer. The GN and LN are the frame-wise group normalization and layer normalization. The GELU is the Gaussian error linear unit activation function. The causal PE is the causal learnable positional encoding. The MHA is the multi-head attention layer.
• Figure 4: Speech separation training pipeline that incorporates CSP frontends with diverse speech separation models. Here $\boldsymbol{m}$ is the two-speaker synthetic mixture. $\mathbf{E}_m$ is auditory features of mixture waveform. $\mathbf{\tilde{E}}_m$ is the predictive features. $\hat{\mathbf{M}}_1$ and $\hat{\mathbf{M}}_2$ are predicted masks for two speakers, respectively, and $\hat{\boldsymbol{s}}_1$, $\hat{\boldsymbol{s}}_2$ are two reconstructed target reference speech.
• Figure 5: The histogram of SI-SDRi (dB) on individual utterances. The red bar is the number of samples for special SI-SDRi (dB) using the causal ConvTasNet model. The blue bar on the causal ConvTasNet with our proposed CSP frontend in Sys.11.