Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoupled Spatial and Temporal Processing for Resource Efficient Multichannel Speech Enhancement

Ashutosh Pandey, Buye Xu

TL;DR

The paper tackles resource-efficient multichannel speech enhancement in the time domain, aiming for ultra-low latency and reduced compute. It introduces a decoupled spatial-temporal architecture: spatial convolution learns multiple frequency-dependent spatial filters to produce multichannel representations, while a single-channel LSTM refines temporal content which is then propagated to other channels via elementwise multiplication; dense connections across blocks further improve learning. The training relies on a phase-constrained magnitude loss to balance speech and interference reconstruction. Empirical results on the DNS Challenge show that the proposed D-LL-RNN family achieves strong enhancement with far fewer GFLOPs and parameters, while achieving a latency of $L_o=32$ samples, i.e., $2~\text{ms}$, enabling practical real-time deployment.

Abstract

We present a novel model designed for resource-efficient multichannel speech enhancement in the time domain, with a focus on low latency, lightweight, and low computational requirements. The proposed model incorporates explicit spatial and temporal processing within deep neural network (DNN) layers. Inspired by frequency-dependent multichannel filtering, our spatial filtering process applies multiple trainable filters to each hidden unit across the spatial dimension, resulting in a multichannel output. The temporal processing is applied over a single-channel output stream from the spatial processing using a Long Short-Term Memory (LSTM) network. The output from the temporal processing stage is then further integrated into the spatial dimension through elementwise multiplication. This explicit separation of spatial and temporal processing results in a resource-efficient network design. Empirical findings from our experiments show that our proposed model significantly outperforms robust baseline models while demanding far fewer parameters and computations, while achieving an ultra-low algorithmic latency of just 2 milliseconds.

Decoupled Spatial and Temporal Processing for Resource Efficient Multichannel Speech Enhancement

TL;DR

The paper tackles resource-efficient multichannel speech enhancement in the time domain, aiming for ultra-low latency and reduced compute. It introduces a decoupled spatial-temporal architecture: spatial convolution learns multiple frequency-dependent spatial filters to produce multichannel representations, while a single-channel LSTM refines temporal content which is then propagated to other channels via elementwise multiplication; dense connections across blocks further improve learning. The training relies on a phase-constrained magnitude loss to balance speech and interference reconstruction. Empirical results on the DNS Challenge show that the proposed D-LL-RNN family achieves strong enhancement with far fewer GFLOPs and parameters, while achieving a latency of samples, i.e., , enabling practical real-time deployment.

Abstract

We present a novel model designed for resource-efficient multichannel speech enhancement in the time domain, with a focus on low latency, lightweight, and low computational requirements. The proposed model incorporates explicit spatial and temporal processing within deep neural network (DNN) layers. Inspired by frequency-dependent multichannel filtering, our spatial filtering process applies multiple trainable filters to each hidden unit across the spatial dimension, resulting in a multichannel output. The temporal processing is applied over a single-channel output stream from the spatial processing using a Long Short-Term Memory (LSTM) network. The output from the temporal processing stage is then further integrated into the spatial dimension through elementwise multiplication. This explicit separation of spatial and temporal processing results in a resource-efficient network design. Empirical findings from our experiments show that our proposed model significantly outperforms robust baseline models while demanding far fewer parameters and computations, while achieving an ultra-low algorithmic latency of just 2 milliseconds.
Paper Structure (13 sections, 3 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 13 sections, 3 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model Description
  3. Problem Formulation
  4. Spatial Convolution
  5. Spatio-Temporal Block
  6. Model Architecture
  7. Loss Function
  8. Experiments
  9. Dataset
  10. Experimental Settings
  11. Baseline Models
  12. Experimental Results
  13. Conclusions

Figures (3)

  • Figure 1: A spatial convolution layer with $S_{i}$ input channels, $S_{o}$ output channels and $F$ hidden units.
  • Figure 2: Spatio-temporal Block.
  • Figure 3: The proposed model for time-domain multichannel speech enhancement.