Channel-Combination Algorithms for Robust Distant Voice Activity and Overlapped Speech Detection

TL;DR

The paper presents a unified, multi-channel approach to robust distant VAD and OSD for speaker diarization, centered on Self-Attention Channel Combinator (SACC) front-ends and their complex-domain variants. It introduces STFT-based, analytic-filter, and complex Beampattern-enabled SACC architectures, as well as a channel-number invariant loss to mitigate array-mismatch issues. Empirical results on the AMI corpus show that complex SACC variants (EcSACC/IcSACC) deliver strong segmentation and diarization performance, with beampattern analyses offering explainability of the learned spatial filters. The invariant training substantially improves robustness to changes in the number and configuration of microphones, enabling consistent OSD and competitive DER/JER across array setups. Overall, the work advances robust, explainable multichannel VAD+OSD for distant-speech diarization and highlights directions for cross-array generalization and spatial-feature integration.

Abstract

Voice Activity Detection (VAD) and Overlapped Speech Detection (OSD) are key pre-processing tasks for speaker diarization. In the meeting context, it is often easier to capture speech with a distant device. This consideration however leads to severe performance degradation. We study a unified supervised learning framework to solve distant multi-microphone joint VAD and OSD (VAD+OSD). This paper investigates various multi-channel VAD+OSD front-ends that weight and combine incoming channels. We propose three algorithms based on the Self-Attention Channel Combinator (SACC), previously proposed in the literature. Experiments conducted on the AMI meeting corpus exhibit that channel combination approaches bring significant VAD+OSD improvements in the distant speech scenario. Specifically, we explore the use of learned complex combination weights and demonstrate the benefits of such an approach in terms of explainability. Channel combination-based VAD+OSD systems are evaluated on the final back-end task, i.e. speaker diarization, and show significant improvements. Finally, since multi-channel systems are trained given a fixed array configuration, they may fail in generalizing to other array set-ups, e.g. mismatched number of microphones. A channel-number invariant loss is proposed to learn a unique feature representation regardless of the number of available microphones. The evaluation conducted on mismatched array configurations highlights the robustness of this training strategy.

Figures (6)

• Figure 1: Simplified sequence-to-sequence model architecture for joint VAD+OSD. The feature extraction is different between the single channel $C=1$ and the multi-channel $C>1$ scenario. The sequence modeling step remains unchanged regardless the number of channels.
• Figure 2: Channel combination procedure for multi-channel feature extraction. The time-frequency representation and the weight calculation algorithm depend on the algorithm considered (e.g. SACC).
• Figure 3: Weight calculation procedures, with (b) to (d) being the proposed methods. (a) STFT-based SACC gong_self-attention_2021, (b) SACC based on the analytic filter bank, (c) EcSACC, and (d) IcSACC. For complex systems (c) and (d), magnitude and phase ($\|\cdot\|$-$\phi$) can be replaced by real and imaginary parts ($\mathfrak{R}$-$\mathfrak{I}$).
• Figure 4: Time-averaged beampatterns (white line) at frequency $f=750$Hz computed on the IcSACC ((a)-(d)) and EcSACC ((e)-(h)) combination weights on 1-second segments from the AMI development set. The heatmap represents the SRP-PHAT acoustic map computed on a 2m radius circular plane. The beampatterns are normalized between 0 and 1.
• Figure 5: Flowchart of the channel-number invariant training procedure. Only one duplicated segment is represented here ($P$=1).