Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

TL;DR

The paper addresses real-time turn-taking prediction in spoken dialogue systems, where traditional silence-based cues yield delays and interruptions. It proposes a Voice Activity Projection (VAP) framework that uses a CPC-based encoder, per-channel self-attention, and a cross-attention transformer to forecast two-speaker voice activity over a 2-s horizon, producing signals such as $p_{now}$ and $p_{future}$ to guide turns. Findings show that the system can operate in real time on CPU hardware with around a 1-s input context without sacrificing accuracy, and multilingual variants for English and Mandarin perform comparably. This work enables continuous, low-latency turn-taking decisions suitable for practical deployment in SDSs and motivates further dialogue-based evaluations.

Abstract

A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.

Figures (3)

• Figure 1: Architecture of the VAP model
• Figure 2: Discretizing bins for the VAP model
• Figure 3: Output example of multilingual VAP - Each graph consists of, from top to bottom, input waveforms of both participants, near future voiced probability ($p_{now}$), and future voiced probability ($p_{future}$) among participants.