Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising

Yoto Fujita, Aditya Arie Nugraha, Diego Di Carlo, Yoshiaki Bando, Mathieu Fontaine, Kazuyoshi Yoshii

TL;DR

Speech enhancement for real-time automatic speech recognition (ASR) in real environments is described and weighted power minimization distortionless response (WPD) beamforming, a unified version of WPE and minimum power distortionless response (MPDR), whose joint dereverberation and denoising filter is estimated using a DNN is proposed.

Abstract

This paper describes speech enhancement for realtime automatic speech recognition (ASR) in real environments. A standard approach to this task is to use neural beamforming that can work efficiently in an online manner. It estimates the masks of clean dry speech from a noisy echoic mixture spectrogram with a deep neural network (DNN) and then computes a enhancement filter used for beamforming. The performance of such a supervised approach, however, is drastically degraded under mismatched conditions. This calls for run-time adaptation of the DNN. Although the ground-truth speech spectrogram required for adaptation is not available at run time, blind dereverberation and separation methods such as weighted prediction error (WPE) and fast multichannel nonnegative matrix factorization (FastMNMF) can be used for generating pseudo groundtruth data from a mixture. Based on this idea, a prior work proposed a dual-process system based on a cascade of WPE and minimum variance distortionless response (MVDR) beamforming asynchronously fine-tuned by block-online FastMNMF. To integrate the dereverberation capability into neural beamforming and make it fine-tunable at run time, we propose to use weighted power minimization distortionless response (WPD) beamforming, a unified version of WPE and minimum power distortionless response (MPDR), whose joint dereverberation and denoising filter is estimated using a DNN. We evaluated the impact of run-time adaptation under various conditions with different numbers of speakers, reverberation times, and signal-to-noise ratios (SNRs).

Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising

TL;DR

Speech enhancement for real-time automatic speech recognition (ASR) in real environments is described and weighted power minimization distortionless response (WPD) beamforming, a unified version of WPE and minimum power distortionless response (MPDR), whose joint dereverberation and denoising filter is estimated using a DNN is proposed.

Abstract

This paper describes speech enhancement for realtime automatic speech recognition (ASR) in real environments. A standard approach to this task is to use neural beamforming that can work efficiently in an online manner. It estimates the masks of clean dry speech from a noisy echoic mixture spectrogram with a deep neural network (DNN) and then computes a enhancement filter used for beamforming. The performance of such a supervised approach, however, is drastically degraded under mismatched conditions. This calls for run-time adaptation of the DNN. Although the ground-truth speech spectrogram required for adaptation is not available at run time, blind dereverberation and separation methods such as weighted prediction error (WPE) and fast multichannel nonnegative matrix factorization (FastMNMF) can be used for generating pseudo groundtruth data from a mixture. Based on this idea, a prior work proposed a dual-process system based on a cascade of WPE and minimum variance distortionless response (MVDR) beamforming asynchronously fine-tuned by block-online FastMNMF. To integrate the dereverberation capability into neural beamforming and make it fine-tunable at run time, we propose to use weighted power minimization distortionless response (WPD) beamforming, a unified version of WPE and minimum power distortionless response (MPDR), whose joint dereverberation and denoising filter is estimated using a DNN. We evaluated the impact of run-time adaptation under various conditions with different numbers of speakers, reverberation times, and signal-to-noise ratios (SNRs).

Paper Structure

This paper contains 15 sections, 22 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dereverberation
  4. Speech Enhancement
  5. Joint Speech Dereverberation and Denoising
  6. Blind Source Separation
  7. Proposed method
  8. Joint Neural Speech Dereverberation and Denoising
  9. Pretraining of Mask Estimator
  10. Run-Time Adaptation of Mask Estimator
  11. Evaluation
  12. Dataset
  13. Experimental Settings
  14. Experimental Results
  15. Conclusion

Figures (2)

  • Figure 1: The overview of our proposed adaptation of neural beamforming based on blind dereverberation by WPE and blind source separation by FastMNMF.
  • Figure 2: The evaluation result of the joint adaptation of dereverberation and denoising by mask-based WPD beamforming based on blind dereverberation by WPE and BSS by FastMNMF. Blue lines refer to the "default" setting, constant across different plots. The shaded regions surrounding each line indicate the 95% confidence intervals.