Gaze-Enhanced Multimodal Turn-Taking Prediction in Triadic Conversations

TL;DR

This work tackles turn-taking prediction in triadic conversations by introducing a lightweight framework that fuses egocentric gaze with spatial speaker localization and binary voice activity (VAD). It presents two models: a single-user version focusing on the target's cues and a multi-user version with an attention mechanism to capture interactions among all participants. Using a Reality Labs CHAT-derived dataset collected with Aria glasses, the approach demonstrates that gaze, especially when combined with VAD and spatial cues, improves role- and behavior-based turn-taking predictions over a VAD-only baseline, with multi-user gaze providing the strongest gains. The solution is privacy-conscious and computationally efficient, offering practical benefits for adaptive beamforming and hearing assistance in smart glasses in noisy environments.

Abstract

Turn-taking prediction is crucial for seamless interactions. This study introduces a novel, lightweight framework for accurate turn-taking prediction in triadic conversations without relying on computationally intensive methods. Unlike prior approaches that either disregard gaze or treat it as a passive signal, our model integrates gaze with speaker localization, structuring it within a spatial constraint to transform it into a reliable predictive cue. Leveraging egocentric behavioral cues, our experiments demonstrate that incorporating gaze data from a single-user significantly improves prediction performance, while gaze data from multiple-users further enhances it by capturing richer conversational dynamics. This study presents a lightweight and privacy-conscious approach to support adaptive, directional sound control, enhancing speech intelligibility in noisy environments, particularly for hearing assistance in smart glasses.

Figures (4)

• Figure 1: Gaze behavior during turn transitions in triadic conversations is shown in peri-stimulus time histogram, with gaze azimuth coordinates around the turn-transition moment (dotted vertical lines), accumulating data around these events. This visualization is based on one session from our collected dataset, spanning 1 second before and after the transition. Horizontal bold colored lines indicate other users' positions. For detailed descriptions of (a)-(c), please refer to the main text.
• Figure 2: Architecture of the single- and multi-user models. (a) The single-user model only uses the target user's feature. (b) The multi-user model incorporates features from all users with a user attention mechanism to capture the dynamics multi-user interactions.
• Figure 3: Confusion matrices of Table 1.
• Figure 4: A 10-second video segment illustrating turn-taking behavior for target user A and reference users B and C. Panels (a) and (b) present role-based and behavior-based classifications, respectively, with color-coded turn-taking labels when voice is detected. The first 3 rows display ground truth labels for A, B and C, while the next 3 rows display predictions from the binary VAD, single-user model, and multi-user model for target user A. Gaze direction (red dots) and gaze shifts (white arrows) provide key turn-taking cues. For detailed descriptions of (a.1)–(b.3), please refer to the main text.