Multichannel Long-Term Streaming Neural Speech Enhancement for Static and Moving Speakers

TL;DR

This work advances multichannel speech enhancement by converting a powerful offline SpatialNet into online, streaming variants capable of long-term temporal-spatial modeling for static and moving speakers. It introduces three online narrow-band architectures—Masked Self-Attention, Retention, and Mamba—along with a short-signal training plus long-signal fine-tuning strategy to improve length extrapolation. Experiments on simulated environments show that the online methods, particularly Mamba, deliver strong speech enhancement with linear inference complexity, though the offline SpatialNet remains superior when future context is available. The authors also provide open-source implementations to enable practical adoption and reproducibility.

Abstract

In this work, we extend our previously proposed offline SpatialNet for long-term streaming multichannel speech enhancement in both static and moving speaker scenarios. SpatialNet exploits spatial information, such as the spatial/steering direction of speech, for discriminating between target speech and interferences, and achieved outstanding performance. The core of SpatialNet is a narrow-band self-attention module used for learning the temporal dynamic of spatial vectors. Towards long-term streaming speech enhancement, we propose to replace the offline self-attention network with online networks that have linear inference complexity w.r.t signal length and meanwhile maintain the capability of learning long-term information. Three variants are developed based on (i) masked self-attention, (ii) Retention, a self-attention variant with linear inference complexity, and (iii) Mamba, a structured-state-space-based RNN-like network. Moreover, we investigate the length extrapolation ability of different networks, namely test on signals that are much longer than training signals, and propose a short-signal training plus long-signal fine-tuning strategy, which largely improves the length extrapolation ability of the networks within limited training time. Overall, the proposed online SpatialNet achieves outstanding speech enhancement performance for long audio streams, and for both static and moving speakers. The proposed method is open-sourced in https://github.com/Audio-WestlakeU/NBSS.

Figures (2)

• Figure 1: Network structure of SpatialNet and three variants of oSpatialNet.
• Figure 2: Speech enhancement performance for different training strategies.