EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild

TL;DR

EgoSpeak tackles the problem of predicting when an egocentric conversational agent should speak in real-world settings. By processing untrimmed first-person video and audio in real time, the framework predicts a speak-probability for upcoming moments, enabling proactive turn-taking. The approach integrates first-person RGB, audio cues, online processing, and untrimmed video streams, and is pretrained with the large-scale YT-Conversation dataset to boost multimodal understanding. Evaluations on EasyCom and Ego4D show that EgoSpeak outperforms baselines and that motion signals and multimodal fusion play critical roles, with modest gains from pretraining. The work contributes a practical, real-time turn-taking solution for in-the-wild scenarios and provides avenues for further enhancements via end-to-end learning and domain adaptation.

Abstract

Predicting when to initiate speech in real-world environments remains a fundamental challenge for conversational agents. We introduce EgoSpeak, a novel framework for real-time speech initiation prediction in egocentric streaming video. By modeling the conversation from the speaker's first-person viewpoint, EgoSpeak is tailored for human-like interactions in which a conversational agent must continuously observe its environment and dynamically decide when to talk. Our approach bridges the gap between simplified experimental setups and complex natural conversations by integrating four key capabilities: (1) first-person perspective, (2) RGB processing, (3) online processing, and (4) untrimmed video processing. We also present YT-Conversation, a diverse collection of in-the-wild conversational videos from YouTube, as a resource for large-scale pretraining. Experiments on EasyCom and Ego4D demonstrate that EgoSpeak outperforms random and silence-based baselines in real time. Our results also highlight the importance of multimodal input and context length in effectively deciding when to speak.

Figures (9)

• Figure 1: EgoSpeak models speech initiation in real time from the camera wearer’s (camera icon) egocentric video stream, mirroring how a real-world agent would perceive and engage in dynamic, multi-speaker environments.
• Figure 2: Overview of the EgoSpeak framework. At each time step, the model processes an untrimmed egocentric video and audio stream, classifying them in real time into three categories: background (no speech), other person speaking, and target speaker (camera wearer) speaking. These probabilities are visualized at the bottom, where the model anticipates near-future frames and enables proactive speech initiation for conversational agents.
• Figure 3: Converting Transcript to Per-Frame Labels. Colors indicate: gray - background, orange - target speaker speaking, purple - other speaker speaking. Labels are one-hot encoded for classification.
• Figure 4: Sample frames from YT-Conversation dataset. The dataset includes a diverse range of conversational scenarios from YouTube, such as podcasts, interviews, and informal dialogues, representing various real-world conversation formats.
• Figure 5: Video duration distribution for YT-Conversation. Our online formulation allows the use of long video clips, some even exceeding 900 seconds.