Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement

Robert Sutherland, George Close, Thomas Hain, Stefan Goetze, Jon Barker

TL;DR

This work shows that distances between encoder representations from the large self-supervised model WavLM correlate strongly with speech intelligibility and can serve as a loss for training a denoising stage in hearing aids. By integrating a WavLM-based distance into the training objective, the authors achieve better HASPI, STOI, PESQ, and SI-SNR than a traditional SNR-based loss, while keeping the final denoiser small and efficient for real-time use. The method leverages training-time access to a big model, with a two-stage hearing-aid pipeline (denoiser and amplifier) and a differentiable hearing-loss simulator to optimize perceptual outcomes. On CPC2 and hearing-aid datasets (CEC1/CEC2), the approach yields meaningful intelligibility gains, suggesting practical impact for deployable hearing-aid systems that must balance performance and hardware constraints.

Abstract

Machine learning techniques are an active area of research for speech enhancement for hearing aids, with one particular focus on improving the intelligibility of a noisy speech signal. Recent work has shown that feature encodings from self-supervised speech representation models can effectively capture speech intelligibility. In this work, it is shown that the distance between self-supervised speech representations of clean and noisy speech correlates more strongly with human intelligibility ratings than other signal-based metrics. Experiments show that training a speech enhancement model using this distance as part of a loss function improves the performance over using an SNR-based loss function, demonstrated by an increase in HASPI, STOI, PESQ and SI-SNR scores. This method takes inference of a high parameter count model only at training time, meaning the speech enhancement model can remain smaller, as is required for hearing aids.

Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement

TL;DR

This work shows that distances between encoder representations from the large self-supervised model WavLM correlate strongly with speech intelligibility and can serve as a loss for training a denoising stage in hearing aids. By integrating a WavLM-based distance into the training objective, the authors achieve better HASPI, STOI, PESQ, and SI-SNR than a traditional SNR-based loss, while keeping the final denoiser small and efficient for real-time use. The method leverages training-time access to a big model, with a two-stage hearing-aid pipeline (denoiser and amplifier) and a differentiable hearing-loss simulator to optimize perceptual outcomes. On CPC2 and hearing-aid datasets (CEC1/CEC2), the approach yields meaningful intelligibility gains, suggesting practical impact for deployable hearing-aid systems that must balance performance and hardware constraints.

Abstract

Machine learning techniques are an active area of research for speech enhancement for hearing aids, with one particular focus on improving the intelligibility of a noisy speech signal. Recent work has shown that feature encodings from self-supervised speech representation models can effectively capture speech intelligibility. In this work, it is shown that the distance between self-supervised speech representations of clean and noisy speech correlates more strongly with human intelligibility ratings than other signal-based metrics. Experiments show that training a speech enhancement model using this distance as part of a loss function improves the performance over using an SNR-based loss function, demonstrated by an increase in HASPI, STOI, PESQ and SI-SNR scores. This method takes inference of a high parameter count model only at training time, meaning the speech enhancement model can remain smaller, as is required for hearing aids.
Paper Structure (13 sections, 3 equations, 4 figures, 2 tables)

This paper contains 13 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Self-Supervised Speech Representations for Speech Enhancement
  3. WavLM Representation Distances
  4. The 2nd Clarity Prediction Challenge (CPC2) Dataset
  5. Correlation Analysis
  6. Hearing Aid System
  7. Denoiser
  8. Amplifier
  9. Experiment Setup
  10. Clarity Enhancement Challenge Datasets
  11. Training Setup
  12. Results
  13. Conclusion

Figures (4)

  • Figure 1: Representations extracted from WavLM model stages.
  • Figure 2: Distribution of human intelligibility labels in the CPC2 train set
  • Figure 3: Correlations of $-\mathcal{L}_\mathrm{SNR}$, $-\mathcal{L}_\mathrm{WLM}$, HASPI and STOI with the human intelligibility labels for the CPC2 dataset.
  • Figure 4: Workflow of the hearing aid system. For a $C$-channel signal $\mathbf{x}=\left[x_0,\dots,x_C\right]$, a reference channel $x_0$ is input to the spectral encoder, while the spatial encoder takes all $C$ channels as input.