Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Blind Source Separation and Diarization for Distant Speech Recognition

Yoshiaki Bando, Tomohiko Nakamura, Shinji Watanabe

TL;DR

This work tackles distant speech recognition with unknown and time-varying numbers of active speakers by proposing a weakly supervised neural approach that jointly performs speech separation and diarization. The method, neural FCASA, extends neural FCA to estimate latent source features and dynamic speaker activity while learning diagonalized-space PSDs via a neural inference model, trained with a multi-task objective that combines an ELBO-based separation term and a supervised diarization term. It achieves competitive and superior WERs compared with GSS using oracle diarization on AMI, and provides diarization outputs without external speaker activity information. The approach demonstrates the feasibility of end-to-end frontend enhancement for multi-talker DSR and suggests extensions toward continuous sequential separation and diarization in real-world scenarios.

Abstract

This paper presents a neural method for distant speech recognition (DSR) that jointly separates and diarizes speech mixtures without supervision by isolated signals. A standard separation method for multi-talker DSR is a statistical multichannel method called guided source separation (GSS). While GSS does not require signal-level supervision, it relies on speaker diarization results to handle unknown numbers of active speakers. To overcome this limitation, we introduce and train a neural inference model in a weakly-supervised manner, employing the objective function of a statistical separation method. This training requires only multichannel mixtures and their temporal annotations of speaker activities. In contrast to GSS, the trained model can jointly separate and diarize speech mixtures without any auxiliary information. The experiments with the AMI corpus show that our method outperforms GSS with oracle diarization results regarding word error rates. The code is available online.

Neural Blind Source Separation and Diarization for Distant Speech Recognition

TL;DR

This work tackles distant speech recognition with unknown and time-varying numbers of active speakers by proposing a weakly supervised neural approach that jointly performs speech separation and diarization. The method, neural FCASA, extends neural FCA to estimate latent source features and dynamic speaker activity while learning diagonalized-space PSDs via a neural inference model, trained with a multi-task objective that combines an ELBO-based separation term and a supervised diarization term. It achieves competitive and superior WERs compared with GSS using oracle diarization on AMI, and provides diarization outputs without external speaker activity information. The approach demonstrates the feasibility of end-to-end frontend enhancement for multi-talker DSR and suggests extensions toward continuous sequential separation and diarization in real-world scenarios.

Abstract

This paper presents a neural method for distant speech recognition (DSR) that jointly separates and diarizes speech mixtures without supervision by isolated signals. A standard separation method for multi-talker DSR is a statistical multichannel method called guided source separation (GSS). While GSS does not require signal-level supervision, it relies on speaker diarization results to handle unknown numbers of active speakers. To overcome this limitation, we introduce and train a neural inference model in a weakly-supervised manner, employing the objective function of a statistical separation method. This training requires only multichannel mixtures and their temporal annotations of speaker activities. In contrast to GSS, the trained model can jointly separate and diarize speech mixtures without any auxiliary information. The experiments with the AMI corpus show that our method outperforms GSS with oracle diarization results regarding word error rates. The code is available online.
Paper Structure (14 sections, 18 equations, 3 figures, 1 table)

This paper contains 14 sections, 18 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Blind source separation
  4. Neural full-rank spatial covariance analysis
  5. Joint Speech Separation and Diarization Based on Neural FastFCA
  6. Generative model of multichannel mixture signals
  7. Inference model
  8. Training without isolated source signals
  9. Experimental Evaluation
  10. Dataset
  11. Experimental configurations
  12. Experimental results
  13. Conclusion
  14. Acknowledgement

Figures (3)

  • Figure 1: The overview of our joint separation and diarization.
  • Figure 2: The inference model of neural FastFCA, which employs the hybrid architecture proposed in Scheibler2021.
  • Figure 3: Examples of separation results by Neural FCA and Neural FCASA. Neural FCA over-separated one source into third and fourth results.