Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise-Aware Speech Separation with Contrastive Learning

Zizheng Zhang, Chen Chen, Hsin-Hung Chen, Xiang Liu, Yuchen Hu, Eng Siong Chng

TL;DR

Noisy speech separation remains challenging due to leakage of background noise into target speakers. The authors propose Noise-Aware Speech Separation (NASS), which adds an auxiliary noise output and a patch-wise contrastive learning objective to minimize mutual information between noise and separated speech within a conventional encoder–separator–decoder framework. The approach includes detailed mechanisms for an additional noise pathway (ANO) and a patch-based MI minimization via cosine-similarity contrastive loss, achieving $1$–$2$ dB improvements in $SI$-SNR$\!i$/$SDR$-$i$ over strong baselines (DPRNN and SepFormer) on WHAM! and LibriMix with only a small parameter overhead. These results demonstrate improved speech quality under noisy conditions and suggest that noise-aware supervision can be extended to other robust speech models and tasks.

Abstract

Recently, speech separation (SS) task has achieved remarkable progress driven by deep learning technique. However, it is still challenging to separate target speech from noisy mixture, as the neural model is vulnerable to assign background noise to each speaker. In this paper, we propose a noise-aware SS (NASS) method, which aims to improve the speech quality for separated signals under noisy conditions. Specifically, NASS views background noise as an additional output and predicts it along with other speakers in a mask-based manner. To effectively denoise, we introduce patch-wise contrastive learning (PCL) between noise and speaker representations from the decoder input and encoder output. PCL loss aims to minimize the mutual information between predicted noise and other speakers at multiple-patch level to suppress the noise information in separated signals. Experimental results show that NASS achieves 1 to 2dB SI-SNRi or SDRi over DPRNN and Sepformer on WHAM! and LibriMix noisy datasets, with less than 0.1M parameter increase.

Noise-Aware Speech Separation with Contrastive Learning

TL;DR

Noisy speech separation remains challenging due to leakage of background noise into target speakers. The authors propose Noise-Aware Speech Separation (NASS), which adds an auxiliary noise output and a patch-wise contrastive learning objective to minimize mutual information between noise and separated speech within a conventional encoder–separator–decoder framework. The approach includes detailed mechanisms for an additional noise pathway (ANO) and a patch-based MI minimization via cosine-similarity contrastive loss, achieving – dB improvements in -SNR/- over strong baselines (DPRNN and SepFormer) on WHAM! and LibriMix with only a small parameter overhead. These results demonstrate improved speech quality under noisy conditions and suggest that noise-aware supervision can be extended to other robust speech models and tasks.

Abstract

Recently, speech separation (SS) task has achieved remarkable progress driven by deep learning technique. However, it is still challenging to separate target speech from noisy mixture, as the neural model is vulnerable to assign background noise to each speaker. In this paper, we propose a noise-aware SS (NASS) method, which aims to improve the speech quality for separated signals under noisy conditions. Specifically, NASS views background noise as an additional output and predicts it along with other speakers in a mask-based manner. To effectively denoise, we introduce patch-wise contrastive learning (PCL) between noise and speaker representations from the decoder input and encoder output. PCL loss aims to minimize the mutual information between predicted noise and other speakers at multiple-patch level to suppress the noise information in separated signals. Experimental results show that NASS achieves 1 to 2dB SI-SNRi or SDRi over DPRNN and Sepformer on WHAM! and LibriMix noisy datasets, with less than 0.1M parameter increase.
Paper Structure (18 sections, 9 equations, 3 figures, 4 tables)

This paper contains 18 sections, 9 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. NASS Method
  3. Mask-based Architecture
  4. Encoder and Decoder
  5. Masking Net and Additional Noise Output
  6. Patch-wise Contrastive Learning
  7. Sampler and Reshaper
  8. Contrastive Loss
  9. Training Objective
  10. Experiments
  11. Datasets
  12. Experimental Setup
  13. Metrics and Baselines
  14. Results
  15. Effect of Additional Noise Output
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The overall pipeline of NASS. $x_n$ and $\hat{n}$ denote the noisy input and predicted noise. $\hat{s}_1$ and $\hat{s}_2$ are separated speech while $s_1$ and $s_2$ are the ground-truth. $h_{\hat{s}_1}$, $h_{\hat{s}_2}$ and $h_{\hat{n}}$ in dashed box are predicted representations, while $h_{s_1}$ and $h_{s_2}$ in solid box are the ground-truth. "P" denotes the mutual information between separated and ground-truth speech is maximized while "N" denotes the mutual information between separated speech and noise is minimized.
  • Figure 2: The illustration of patch-wise contrastive learning. For the $i$-th sampling of $K$ times, one query example $r^i_q$, positive example $r^i_p$ and $M$ negative examples ${r_n^{i,j}}$ ($j \in [1,M]$) are sampled from predicted speech representation $h_{\hat{s}_a}$, ground-truth speech representation $h_{s_a}$ and predicted noise representation $h_{\hat{n}}$, respectively, "CS" denotes cosine similarity.
  • Figure 3: Spectrum results on Libri2mix with Sepformer. Subplot (a) denotes the mixture; (b), (c) are baseline results; (d), (e), (f) are NASS results. Note that (d) is the noise output.