Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

All Neural Low-latency Directional Speech Extraction

Ashutosh Pandey, Sanha Lee, Juan Azcarreta, Daniel Wong, Buye Xu

TL;DR

This paper tackles low-latency directional speech extraction by introducing a fully neural framework that uses trainable DOA embeddings drawn from a predefined spatial grid to condition an all-neural, time-domain speech extractor called the Directional Recurrent Network (DRN). The method integrates two forms of DOA embeddings—channel-wise and frame-wise—into the spatial and temporal processing via a fusion scheme, achieving end-to-end optimization with a speech enhancement objective and millisecond-scale latency. Key findings show that DOA embeddings, especially azimuth-elevation, improve performance and robustness to DOA mismatch, with rapid adaptation to abrupt DOA switches in dynamic scenes; the approach also outperforms several strong baselines under low-latency constraints. The work advances practical directional speech extraction for multichannel setups, enabling fast, robust performance in scenarios like augmented reality and wearable devices, and points to future work on moving-source training and more dynamic motion handling.

Abstract

We introduce a novel all neural model for low-latency directional speech extraction. The model uses direction of arrival (DOA) embeddings from a predefined spatial grid, which are transformed and fused into a recurrent neural network based speech extraction model. This process enables the model to effectively extract speech from a specified DOA. Unlike previous methods that relied on hand-crafted directional features, the proposed model trains DOA embeddings from scratch using speech enhancement loss, making it suitable for low-latency scenarios. Additionally, it operates at a high frame rate, taking in DOA with each input frame, which brings in the capability of quickly adapting to changing scene in highly dynamic real-world scenarios. We provide extensive evaluation to demonstrate the model's efficacy in directional speech extraction, robustness to DOA mismatch, and its capability to quickly adapt to abrupt changes in DOA.

All Neural Low-latency Directional Speech Extraction

TL;DR

This paper tackles low-latency directional speech extraction by introducing a fully neural framework that uses trainable DOA embeddings drawn from a predefined spatial grid to condition an all-neural, time-domain speech extractor called the Directional Recurrent Network (DRN). The method integrates two forms of DOA embeddings—channel-wise and frame-wise—into the spatial and temporal processing via a fusion scheme, achieving end-to-end optimization with a speech enhancement objective and millisecond-scale latency. Key findings show that DOA embeddings, especially azimuth-elevation, improve performance and robustness to DOA mismatch, with rapid adaptation to abrupt DOA switches in dynamic scenes; the approach also outperforms several strong baselines under low-latency constraints. The work advances practical directional speech extraction for multichannel setups, enabling fast, robust performance in scenarios like augmented reality and wearable devices, and points to future work on moving-source training and more dynamic motion handling.

Abstract

We introduce a novel all neural model for low-latency directional speech extraction. The model uses direction of arrival (DOA) embeddings from a predefined spatial grid, which are transformed and fused into a recurrent neural network based speech extraction model. This process enables the model to effectively extract speech from a specified DOA. Unlike previous methods that relied on hand-crafted directional features, the proposed model trains DOA embeddings from scratch using speech enhancement loss, making it suitable for low-latency scenarios. Additionally, it operates at a high frame rate, taking in DOA with each input frame, which brings in the capability of quickly adapting to changing scene in highly dynamic real-world scenarios. We provide extensive evaluation to demonstrate the model's efficacy in directional speech extraction, robustness to DOA mismatch, and its capability to quickly adapt to abrupt changes in DOA.
Paper Structure (16 sections, 2 equations, 4 figures, 3 tables)

This paper contains 16 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Problem formulation
  4. Model architecture
  5. Channel-wise DOA embeddings
  6. Frame-wise DOA embeddings
  7. Fusing DOA embeddings
  8. Dataset Generation
  9. Results
  10. Experiments design
  11. Finding the optimal location embeddings
  12. Higher latency and frequency domain processing
  13. Switching the direction of the target speech
  14. Baseline models
  15. Robustness to DOA Mismatch
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Schematic diagram of DRN.
  • Figure 2: Comparing channel-wise and frame-wise embeddings. In plots labels, A and AE respectively represents azimuth-only and azimuth-elevation embeddings, whereas the number represents hidden size $H$.
  • Figure 3: The spectrograms of a) the noisy audio, b) the DRN enhanced audio and c) the target audio. The input DOA of different talkers are switched to extract corresponding switched talkers at output.
  • Figure 4: Sensitivity of DRN to DOA mismatch.