Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI

Karim Helwani, Hoang Do, James Luan, Sriram Srinivasan

Abstract

We present a real-time front-end for voice-based conversational AI to enable natural turn-taking in two-speaker scenarios by combining primary speaker segmentation with hierarchical End-of-Turn (EOT) detection. To operate robustly in multi-speaker environments, the system continuously identifies and tracks the primary user, ensuring that downstream EOT decisions are not confounded by background conversations. The tracked activity segments are fed to a hierarchical, causal EOT model that predicts the immediate conversational state by independently analyzing per-speaker speech features from both the primary speaker and the bot. Simultaneously, the model anticipates near-future states ($t{+}10/20/30$\,ms) through probabilistic predictions that are aware of the conversation partner's speech. Task-specific knowledge distillation compresses wav2vec~2.0 representations (768\,D) into a compact MFCC-based student (32\,D) for efficient deployment. The system achieves 82\% multi-class frame-level F1 and 70.6\% F1 on Backchannel detection, with 69.3\% F1 on a binary Final vs.\ Others task. On an end-to-end turn-detection benchmark, our model reaches 87.7\% recall vs.\ 58.9\% for Smart Turn~v3 while keeping a median detection latency of 36\,ms versus 800--1300\,ms. Despite using only 1.14\,M parameters, the proposed model matches or exceeds transformer-based baselines while substantially reducing latency and memory footprint, making it suitable for edge deployment.

A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI

Abstract

We present a real-time front-end for voice-based conversational AI to enable natural turn-taking in two-speaker scenarios by combining primary speaker segmentation with hierarchical End-of-Turn (EOT) detection. To operate robustly in multi-speaker environments, the system continuously identifies and tracks the primary user, ensuring that downstream EOT decisions are not confounded by background conversations. The tracked activity segments are fed to a hierarchical, causal EOT model that predicts the immediate conversational state by independently analyzing per-speaker speech features from both the primary speaker and the bot. Simultaneously, the model anticipates near-future states (\,ms) through probabilistic predictions that are aware of the conversation partner's speech. Task-specific knowledge distillation compresses wav2vec~2.0 representations (768\,D) into a compact MFCC-based student (32\,D) for efficient deployment. The system achieves 82\% multi-class frame-level F1 and 70.6\% F1 on Backchannel detection, with 69.3\% F1 on a binary Final vs.\ Others task. On an end-to-end turn-detection benchmark, our model reaches 87.7\% recall vs.\ 58.9\% for Smart Turn~v3 while keeping a median detection latency of 36\,ms versus 800--1300\,ms. Despite using only 1.14\,M parameters, the proposed model matches or exceeds transformer-based baselines while substantially reducing latency and memory footprint, making it suitable for edge deployment.
Paper Structure (20 sections, 2 equations, 3 figures, 1 table)

This paper contains 20 sections, 2 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Endpointing and Turn-taking
  4. Diarization and Primary Speaker Tracking
  5. Relation to VAP and Smart Turn
  6. Primary Speaker Segmentation
  7. Motivation
  8. Feature Encoder (Backbone)
  9. Peeling-based Multi-speaker Embedding Extraction
  10. Online Primary Speaker Clustering
  11. Training Objectives and Augmentation
  12. Performance Evaluation and Model Complexity
  13. Hierarchical End-of-Turn Model
  14. Datasets and Labels
  15. Model Overview
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Detailed overview of the primary speaker segmentation unit. The embeddings are extracted recursively, the gating layer decides how much of the residual signal $r_i$ to be used for a speaker' embedding, the number of speakers in a frame is informed through the stop signal $s_i$. The embeddings are quantized using a soft quantizer into 128 discrete levels.
  • Figure 2: Equal error rate (EER) vs. signal-to-interference ratio (SIR) for primary speaker segmentation. Average EER is $\approx$10%, improving as SIR increases.
  • Figure 3: System Overview.