Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting Consistency-Preserving Loss and Perceptual Contrast Stretching to Boost SSL-based Speech Enhancement

Muhammad Salman Khan, Moreno La Quatra, Kuo-Hsuan Hung, Szu-Wei Fu, Sabato Marco Siniscalchi, Yu Tsao

TL;DR

The paper tackles the insufficient performance of SSL-based speech enhancement by integrating a Conformer-based masking head with a consistency-preserving loss and a perceptual contrast stretching (PCS) pre-processing step. The model employs a WavLM backbone to extract frame-level features and fuses them with compressed STFT magnitudes to predict the Ideal Ratio Mask for enhancement, trained with $wSDR$, mag-L1, and CS-mag-L1 losses. PCS further enhances perceptual quality by weighting STFT magnitudes using a band importance function prior to training. On VoiceBank-DEMAND, the proposed PCS-CS-WavLM achieves state-of-the-art PESQ (e.g., $3.54$) among SSL-based SE methods and shows substantial gains in perceptual metrics, with ablation studies highlighting the importance of the Conformer head and applying PCS to both input and target signals.

Abstract

Self-supervised representation learning (SSL) has attained SOTA results on several downstream speech tasks, but SSL-based speech enhancement (SE) solutions still lag behind. To address this issue, we exploit three main ideas: (i) Transformer-based masking generation, (ii) consistency-preserving loss, and (iii) perceptual contrast stretching (PCS). In detail, conformer layers, leveraging an attention mechanism, are introduced to effectively model frame-level representations and obtain the Ideal Ratio Mask (IRM) for SE. Moreover, we incorporate consistency in the loss function, which processes the input to account for the inconsistency effects of signal reconstruction from the spectrogram. Finally, PCS is employed to improve the contrast of input and target features according to perceptual importance. Evaluated on the VoiceBank-DEMAND task, the proposed solution outperforms previously SSL-based SE solutions when tested on several objective metrics, attaining a SOTA PESQ score of 3.54.

Exploiting Consistency-Preserving Loss and Perceptual Contrast Stretching to Boost SSL-based Speech Enhancement

TL;DR

The paper tackles the insufficient performance of SSL-based speech enhancement by integrating a Conformer-based masking head with a consistency-preserving loss and a perceptual contrast stretching (PCS) pre-processing step. The model employs a WavLM backbone to extract frame-level features and fuses them with compressed STFT magnitudes to predict the Ideal Ratio Mask for enhancement, trained with , mag-L1, and CS-mag-L1 losses. PCS further enhances perceptual quality by weighting STFT magnitudes using a band importance function prior to training. On VoiceBank-DEMAND, the proposed PCS-CS-WavLM achieves state-of-the-art PESQ (e.g., ) among SSL-based SE methods and shows substantial gains in perceptual metrics, with ablation studies highlighting the importance of the Conformer head and applying PCS to both input and target signals.

Abstract

Self-supervised representation learning (SSL) has attained SOTA results on several downstream speech tasks, but SSL-based speech enhancement (SE) solutions still lag behind. To address this issue, we exploit three main ideas: (i) Transformer-based masking generation, (ii) consistency-preserving loss, and (iii) perceptual contrast stretching (PCS). In detail, conformer layers, leveraging an attention mechanism, are introduced to effectively model frame-level representations and obtain the Ideal Ratio Mask (IRM) for SE. Moreover, we incorporate consistency in the loss function, which processes the input to account for the inconsistency effects of signal reconstruction from the spectrogram. Finally, PCS is employed to improve the contrast of input and target features according to perceptual importance. Evaluated on the VoiceBank-DEMAND task, the proposed solution outperforms previously SSL-based SE solutions when tested on several objective metrics, attaining a SOTA PESQ score of 3.54.
Paper Structure (13 sections, 2 equations, 1 figure, 2 tables)

This paper contains 13 sections, 2 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. SSL-based speech enhancement model
  3. Overall architecture
  4. Speech enhancement head
  5. Loss computation
  6. Perceptual contrast stretching - PCS
  7. Dataset & Experimental Setup
  8. Dataset
  9. Experimental Setup
  10. Experimental Results
  11. SSL based results
  12. Ablation studies
  13. Conclusion

Figures (1)

  • Figure 1: Overall architecture of the proposed SSL-based speech enhancement (SE) model. The magnitude of the STFT is indicated with Mag. CS stands for consistent, and PCS indicates perceptural contrast stretching.