A cross-talk robust multichannel VAD model for multiparty agent interactions trained using synthetic re-recordings

TL;DR

This work tackles cross-talk in a four-channel, hands-free multiparty interaction by introducing MPVAD, a cross-talk robust multichannel VAD that predicts per-channel speech activity using joint spectral features. Two architectures are presented—MPVAD-SC and MPVAD-MC—with a late-fusion variant MPVAD-F—trained on synthetic playback/re-recordings to differentiate near-field speech from cross-talk. MPVAD-MC consistently outperforms single-channel and energy-based baselines and reduces insertion errors in downstream ASR when used to filter audio, while meeting real-time constraints. The study demonstrates practical cross-talk rejection for live multiparty systems and outlines directions for target-speaker VAD and multimodal enhancements.

Abstract

In this work, we propose a novel cross-talk rejection framework for a multi-channel multi-talker setup for a live multiparty interactive show. Our far-field audio setup is required to be hands-free during live interaction and comprises four adjacent talkers with directional microphones in the same space. Such setups often introduce heavy cross-talk between channels, resulting in reduced automatic speech recognition (ASR) and natural language understanding (NLU) performance. To address this problem, we propose voice activity detection (VAD) model for all talkers using multichannel information, which is then used to filter audio for downstream tasks. We adopt a synthetic training data generation approach through playback and re-recording for such scenarios, simulating challenging speech overlap conditions. We train our models on this synthetic data and demonstrate that our approach outperforms single-channel VAD models and energy-based multi-channel VAD algorithm in various acoustic environments. In addition to VAD results, we also present multiparty ASR evaluation results to highlight the impact of using our VAD model for filtering audio in downstream tasks by significantly reducing the insertion error.

Figures (4)

• Figure 1: Speaker and microphone layout used in our application and for the re-recording data collection described in Sec. \ref{['subsec:data']}
• Figure 2: Network architecture of proposed MPVAD-MC model
• Figure 3: Channel activity detection accuracy vs. number of talkers (over the entire evaluation set).
• Figure 4: Confusion matrices showing how different models perform on the speaker counting task. The true number of active speakers are presented along rows and columns show estimated number of speakers based on VAD predictions. (a)-(c): Evaluation set A (d)-(f): Evaluation set C