Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video

TL;DR

This work defines the ESTP task for ego-streaming video QA, where an assistant proactively answers evolving questions in sync with continuous egocentric streams. It introduces ESTP-Bench and the ESTP-F1 metric to jointly evaluate answer quality, timing, and temporal precision, and proposes a full pipeline with a data engine (ESTP-Gen), multi-stage training, and a proactive dynamic compression mechanism. The authors present VideoLLM-EyeWO, which outperforms diverse baselines on online and offline benchmarks and demonstrates improved memory efficiency and coherent multi-turn reasoning. The results highlight the feasibility and impact of proactive, just-in-time, and synchronized reasoning for real-time embodied AI systems in egocentric video settings.

Abstract

Envision an AI capable of functioning in human-like settings, moving beyond mere observation to actively understand, anticipate, and proactively respond to unfolding events. Towards this vision, we focus on the innovative task where, given ego-streaming video input, an assistant proactively answers diverse, evolving questions at the opportune moment, while maintaining synchronized perception and reasoning. This task embodies three key properties: (1) Proactive Coherence, (2) Just-in-Time Responsiveness, and (3) Synchronized Efficiency. To evaluate and address these properties, we first introduce ESTP-Bench (Ego Streaming Proactive Benchmark) alongside the ESTP-F1 metric-a novel framework designed for their rigorous assessment. Secondly, we propose a comprehensive technical pipeline to enable models to tackle this challenging task. This pipeline comprises: (1) a data engine, (2) a multi-stage training strategy, and (3) a proactive dynamic compression technique. Our proposed model effectively addresses these critical properties while outperforming multiple baselines across diverse online and offline benchmarks. Project Page:https://zhangyl4.github.io/publications/eyes-wide-open/

Figures (8)

• Figure 1: An illustrative example of the ESTP task. The figure is structured in three layers: the top layer depicts the model's continuous visual processing and decision-making (See and Think), the middle layer shows the real-world egocentric scene with the human's trajectory, and the bottom layer presents the human-model conversation.
• Figure 2: ESTP Triangle of Impossibility shows trade-offs among the three dimensions: Proactive Coherence, Just-in-Time responsiveness, and Efficiency, which are quantified by contextual performance, recall, and FPS.
• Figure 3: (a) Frequency of scenes or activities from which tasks and questions are derived. (b) Proportion of different proactive and question task types.
• Figure 4: Overview of the proposed pipeline. The figure illustrates the three main components: (a) Data Engine (ESTP-Gen), which automatically generates diverse, multi-turn QA data through a three-stage pipeline. (b) Multi-Stage Training Strategy incrementally builds the model from basic responsiveness to proactive just-in-time accuracy, and ultimately to achieving multi-turn coherence, detailed in Section \ref{['sec:training']}. (c) Proactive Dynamic Compression detailed in Section \ref{['sec:compression']}.
• Figure 4: COIN tangCOINLargescaleDataset2019 Benchmark Top-1 Accuracy comparison across different methods.