Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams

TL;DR

Event-VStream reframes long-stream video understanding as an event-centric problem, detecting meaningful state transitions by fusing motion, semantic drift, and predictive cues to form compact event tokens stored in a persistent memory. By updating memory only at semantic boundaries and decoding text solely at those moments, it avoids frame-level redundancy and preserves long-horizon context with real-time latency. The approach yields strong open-world streaming performance (OVOBench-Realtime) and long-horizon stability (Ego4D), approaching the performance of specialized backbones while using a general-purpose LLM backbone, and maintains sub-0.1 s/token throughput over multi-hour streams. These results suggest significant gains in efficiency and coherence for real-time multimodal systems, with potential extensions to audio-visual and multi-scale temporal reasoning.

Abstract

Real-time understanding of long video streams remains challenging for multimodal large language models (VLMs) due to redundant frame processing and rapid forgetting of past context. Existing streaming systems rely on fixed-interval decoding or cache pruning, which either produce repetitive outputs or discard crucial temporal information. We introduce Event-VStream, an event-aware framework that represents continuous video as a sequence of discrete, semantically coherent events. Our system detects meaningful state transitions by integrating motion, semantic, and predictive cues, and triggers language generation only at those boundaries. Each event embedding is consolidated into a persistent memory bank, enabling long-horizon reasoning while maintaining low latency. Across OVOBench-Realtime, and long-form Ego4D evaluations, Event-VStream achieves competitive performance. It improves over a VideoLLM-Online-8B baseline by +10.4 points on OVOBench-Realtime, achieves performance close to Flash-VStream-7B despite using only a general-purpose LLaMA-3-8B text backbone, and maintains around 70% GPT-5 win rate on 2-hour Ego4D streams.

Figures (8)

• Figure 1: Comparison between timewise-uniform processing and our event-centric grouping. Previous streaming models treat every frame equally over time, leading to redundant computation and temporally fragmented context. In contrast, our method dynamically clusters frames into semantically coherent events (A–C), processing and updating memory only when meaningful visual changes occur.
• Figure 2: Overview of the proposed Event-VStream. Our system dynamically groups continuous video frames into semantically coherent events. Each event embedding is compressed and stored in a persistent memory bankzhang2024flashvstreamxiong2025streamchat, enabling efficient long-horizon reasoning and online question answering under streaming conditions. The model integrates motion-based and semantic cues for event aggregation, retrieves relevant event memories, and performs event-driven decoding to maintain temporal coherence.
• Figure 3: (a) Visual embeddings form block structures, revealing event-level recurrence rather than smooth temporal evolution. (b) Temporal redundancy decays nonlinearly with time, reinforcing the need for Event-VStream compression rather than naïve pooling.
• Figure 4: (Top) Motion–semantic correlation curve. (Bottom) Local motion–semantic transition: motion spikes (a) precede semantic drift (c) by $\sim$2s. Motion spikes precede semantic drift by approximately 2s, suggesting that motion can serve as an early cue for event boundaries.
• Figure 5: Frame-wise motion intensity vs. semantic similarity. Motion spikes often precede drops in cosine similarity, indicating that combining motion signals with semantic cues yields more accurate event boundaries.