DualSep: A Light-weight dual-encoder convolutional recurrent network for real-time in-car speech separation

TL;DR

DualSep addresses real-time speech separation in in-car scenarios with distributed microphone arrays under stringent latency and compute constraints. It fuses DSP with neural networks through a lightweight dual-encoder model, using fixed beamforming for efficiency and IVA as a spatial prior, enabling spatial-spectral fusion. The system supports streaming and non-streaming modes and achieves strong performance with only 0.83M parameters and a 0.39x real-time factor on a CPU, effectively separating speech into distinct zones. This approach offers a practical, low-resource solution for in-vehicle voice interfaces, with demos available to demonstrate real-time capabilities.

Abstract

Advancements in deep learning and voice-activated technologies have driven the development of human-vehicle interaction. Distributed microphone arrays are widely used in in-car scenarios because they can accurately capture the voices of passengers from different speech zones. However, the increase in the number of audio channels, coupled with the limited computational resources and low latency requirements of in-car systems, presents challenges for in-car multi-channel speech separation. To migrate the problems, we propose a lightweight framework that cascades digital signal processing (DSP) and neural networks (NN). We utilize fixed beamforming (BF) to reduce computational costs and independent vector analysis (IVA) to provide spatial prior. We employ dual encoders for dual-branch modeling, with spatial encoder capturing spatial cues and spectral encoder preserving spectral information, facilitating spatial-spectral fusion. Our proposed system supports both streaming and non-streaming modes. Experimental results demonstrate the superiority of the proposed system across various metrics. With only 0.83M parameters and 0.39 real-time factor (RTF) on an Intel Core i7 (2.6GHz) CPU, it effectively separates speech into distinct speech zones. Our demos are available at https://honee-w.github.io/DualSep/.

Figures (1)

• Figure 1: Example of placing a figure with experimental results.