Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations

Amit Meghanani, Thomas Hain

TL;DR

This work addresses the problem that fine-tuning SE models with SSL representations using mean squared error can exploit absolute positional information in SSL encodings, hindering generalization in noisy environments. It introduces two position-invariant strategies: (i) SSL-MSE-PAD, which randomly zero-pads the clean waveform to disrupt positional cues, and (ii) SSL-SoftDTW, which applies speed perturbation to the clean signal and uses a differentiable soft-DTW loss to align SE-enhanced and clean SSL representations by content rather than position. Empirically, SSL-SoftDTW yields faster convergence and stronger improvements on downstream tasks (ASR and PR) under unseen noise, while SSL-MSE-PAD provides limited or marginal gains. The findings suggest that addressing positional exploitation during SE fine-tuning improves robustness of SSL-guided speech models, and that similar position-invariance concepts could be beneficial during SSL pre-training or other SSL-loss settings.

Abstract

Integrating front-end speech enhancement (SE) models with self-supervised learning (SSL)-based speech models is effective for downstream tasks in noisy conditions. SE models are commonly fine-tuned using SSL representations with mean squared error (MSE) loss between enhanced and clean speech. However, MSE is prone to exploiting positional embeddings in SSL models, allowing the objective to be minimised through positional correlations instead of content-related information. This work frames the problem as a general limitation of self-supervised representation fine-tuning and investigates it through representation-guided SE. Two strategies are considered: (1) zero-padding, previously explored in SSL pre-training but here examined in the fine-tuning setting, and (2) speed perturbations with a soft-DTW loss. Experiments show that the soft-DTW-based approach achieves faster convergence and improved downstream performance, underscoring the importance of position-invariant fine-tuning in SSL-based speech modelling.

Position-invariant Fine-tuning of Speech Enhancement Models with Self-supervised Speech Representations

TL;DR

This work addresses the problem that fine-tuning SE models with SSL representations using mean squared error can exploit absolute positional information in SSL encodings, hindering generalization in noisy environments. It introduces two position-invariant strategies: (i) SSL-MSE-PAD, which randomly zero-pads the clean waveform to disrupt positional cues, and (ii) SSL-SoftDTW, which applies speed perturbation to the clean signal and uses a differentiable soft-DTW loss to align SE-enhanced and clean SSL representations by content rather than position. Empirically, SSL-SoftDTW yields faster convergence and stronger improvements on downstream tasks (ASR and PR) under unseen noise, while SSL-MSE-PAD provides limited or marginal gains. The findings suggest that addressing positional exploitation during SE fine-tuning improves robustness of SSL-guided speech models, and that similar position-invariance concepts could be beneficial during SSL pre-training or other SSL-loss settings.

Abstract

Integrating front-end speech enhancement (SE) models with self-supervised learning (SSL)-based speech models is effective for downstream tasks in noisy conditions. SE models are commonly fine-tuned using SSL representations with mean squared error (MSE) loss between enhanced and clean speech. However, MSE is prone to exploiting positional embeddings in SSL models, allowing the objective to be minimised through positional correlations instead of content-related information. This work frames the problem as a general limitation of self-supervised representation fine-tuning and investigates it through representation-guided SE. Two strategies are considered: (1) zero-padding, previously explored in SSL pre-training but here examined in the fine-tuning setting, and (2) speed perturbations with a soft-DTW loss. Experiments show that the soft-DTW-based approach achieves faster convergence and improved downstream performance, underscoring the importance of position-invariant fine-tuning in SSL-based speech modelling.
Paper Structure (11 sections, 7 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 11 sections, 7 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Baseline
  4. Positional Perturbation through Random Zero-Padding
  5. Speed Perturbation with Soft-DTW Loss
  6. Experimental Setup
  7. Frontend SE Model Fine-tuning
  8. Evaluation on SUPERB Downstream Tasks
  9. Results and Discussion
  10. Conclusions and Future Work
  11. Acknowledgement

Figures (2)

  • Figure 1: SSL-MSE: pipeline for fine-tuning a frontend SE model using SSL-based speech representations with MSE loss se_ssl.
  • Figure 2: WER (in %) on test-clean + outdoor noise for ASR across training checkpoints. SE frontends are fine-tuned with different objectives: SSL-MSE, SSL-MSE-PAD, and SSL-SoftDTW. Each curve shows the mean of 5 runs.