Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speech Denoising with Auditory Models

Mark R. Saddler, Andrew Francl, Jenelle Feather, Kaizhi Qian, Yang Zhang, Josh H. McDermott

TL;DR

The paper investigates whether deep perceptual losses derived from recognition networks trained on word and environmental sounds can improve speech denoising over traditional waveform reconstruction. It introduces a two_component framework combining recognition_network_based losses with a Wave_U_Net denoiser and evaluates multiple loss families including cochlear_model losses, using extensive human perceptual tests and standard metrics. The key finding is that deep feature losses can enhance denoising relative to waveform_loss baselines, but the same benefits are achievable with a standard multi_channel auditory filter_bank, suggesting that learned features do not yet provide a unique advantage. The results underscore limitations of current perceptual losses and objective metrics, and point to the need for more expressive transforms or perceptual models to yield robust, generalizable improvements in speech enhancement. The work also highlights that perceptual gains may not be fully captured by traditional metrics, motivating the development of perceptually aligned objective measures.

Abstract

Contemporary speech enhancement predominantly relies on audio transforms that are trained to reconstruct a clean speech waveform. The development of high-performing neural network sound recognition systems has raised the possibility of using deep feature representations as 'perceptual' losses with which to train denoising systems. We explored their utility by first training deep neural networks to classify either spoken words or environmental sounds from audio. We then trained an audio transform to map noisy speech to an audio waveform that minimized the difference in the deep feature representations between the output audio and the corresponding clean audio. The resulting transforms removed noise substantially better than baseline methods trained to reconstruct clean waveforms, and also outperformed previous methods using deep feature losses. However, a similar benefit was obtained simply by using losses derived from the filter bank inputs to the deep networks. The results show that deep features can guide speech enhancement, but suggest that they do not yet outperform simple alternatives that do not involve learned features.

Speech Denoising with Auditory Models

TL;DR

The paper investigates whether deep perceptual losses derived from recognition networks trained on word and environmental sounds can improve speech denoising over traditional waveform reconstruction. It introduces a two_component framework combining recognition_network_based losses with a Wave_U_Net denoiser and evaluates multiple loss families including cochlear_model losses, using extensive human perceptual tests and standard metrics. The key finding is that deep feature losses can enhance denoising relative to waveform_loss baselines, but the same benefits are achievable with a standard multi_channel auditory filter_bank, suggesting that learned features do not yet provide a unique advantage. The results underscore limitations of current perceptual losses and objective metrics, and point to the need for more expressive transforms or perceptual models to yield robust, generalizable improvements in speech enhancement. The work also highlights that perceptual gains may not be fully captured by traditional metrics, motivating the development of perceptually aligned objective measures.

Abstract

Contemporary speech enhancement predominantly relies on audio transforms that are trained to reconstruct a clean speech waveform. The development of high-performing neural network sound recognition systems has raised the possibility of using deep feature representations as 'perceptual' losses with which to train denoising systems. We explored their utility by first training deep neural networks to classify either spoken words or environmental sounds from audio. We then trained an audio transform to map noisy speech to an audio waveform that minimized the difference in the deep feature representations between the output audio and the corresponding clean audio. The resulting transforms removed noise substantially better than baseline methods trained to reconstruct clean waveforms, and also outperformed previous methods using deep feature losses. However, a similar benefit was obtained simply by using losses derived from the filter bank inputs to the deep networks. The results show that deep features can guide speech enhancement, but suggest that they do not yet outperform simple alternatives that do not involve learned features.

Paper Structure

This paper contains 23 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Recognition Networks
  4. Recognition Network Architectures
  5. Recognition Network Training
  6. Audio Transforms
  7. Wave-U-Net Architecture
  8. Deep Feature Losses
  9. Cochlear Model Losses
  10. Wave-U-Net Training
  11. Baselines
  12. Evaluation
  13. Human Perceptual Evaluation and Objective Metrics
  14. Results
  15. Deep Feature Losses Yield Improved Denoising
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Schematic of audio transform training.
  • Figure 2: Rated naturalness vs. SNR for speech processed by Wave-U-Nets trained on deep feature losses, in addition to baseline models trained to reconstruct clean speech waveforms, and two versions of a related prior method Germain2019. Error bars plot SEM (across 54 participants).
  • Figure 3: Rated naturalness vs. SNR for speech processed by Wave-U-Nets trained on cochlear model losses with different filter banks (select examples depicted above). Error bars plot SEM (across 105 participants).