StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding

TL;DR

StreamAgent addresses real-time streaming video understanding by integrating anticipatory planning with a memory-aided perception loop. It forecasts future task-relevant events and regions, aligns current observations with those predictions, and engages tool-augmented perception to proactively gather evidence, all coordinated by a planning-driven decision agent. A streaming KV-cache memory mechanism enables selective recall across long videos by combining on-GPU short-term memory with CPU offloaded long-term memory and dynamic, layer-aware retrieval. Results show leading performance on streaming benchmarks with substantially reduced latency compared to baselines, and strong competitive performance on offline long-video tasks, underscoring the practical value of anticipatory planning and memory-efficient retrieval in real-time video understanding. The approach advances real-time, proactive video QA by fusing future-aware reasoning, hierarchical memory, and targeted perception, with potential applications in autonomous driving and intelligent surveillance.

Abstract

Real-time streaming video understanding in domains such as autonomous driving and intelligent surveillance poses challenges beyond conventional offline video processing, requiring continuous perception, proactive decision making, and responsive interaction based on dynamically evolving visual content. However, existing methods rely on alternating perception-reaction or asynchronous triggers, lacking task-driven planning and future anticipation, which limits their real-time responsiveness and proactive decision making in evolving video streams. To this end, we propose a StreamAgent that anticipates the temporal intervals and spatial regions expected to contain future task-relevant information to enable proactive and goal-driven responses. Specifically, we integrate question semantics and historical observations through prompting the anticipatory agent to anticipate the temporal progression of key events, align current observations with the expected future evidence, and subsequently adjust the perception action (e.g., attending to task-relevant regions or continuously tracking in subsequent frames). To enable efficient inference, we design a streaming KV-cache memory mechanism that constructs a hierarchical memory structure for selective recall of relevant tokens, enabling efficient semantic retrieval while reducing the overhead of storing all tokens in the traditional KV-cache. Extensive experiments on streaming and long video understanding tasks demonstrate that our method outperforms existing methods in response accuracy and real-time efficiency, highlighting its practical value for real-world streaming scenarios.

Figures (8)

• Figure 1: Comparison between StreamAgent and existing methods. Prior online methods videollm-onlineqian2025dispider enable proactive interactions but rely on either (a) alternating interaction, causing slow processing, or (b) asynchronous binary triggers, leading to inaccurate responses. In contrast, StreamAgent integrates continuous perception with task-driven planning and future anticipation, enabling proactive identification of key temporal and spatial cues, and supporting efficient asynchronous reaction through the streaming KV-cache.
• Figure 2: Overview of the StreamAgent framework: At each timestep, the system incrementally updates memory based on the current streaming video clip $v_t$ and memory $m_t$ via captioning in a Markov process. Conditioned on the updated memory and current observation, the planner generates multiple future-aware plans through proactive anticipation simulating incoming videos, spanning three perspectives, Reactive, Proactive, and Speculative. These plans are scored via heuristic evaluation (Eq. \ref{['eq:2']}), jointly considering current situational awareness and predicted future utility. The selected plan either triggers asynchronous reactive behavior or initiates proactive information hunting by invoking tools (e.g., zooming in task-relevant regions in subsequent frames), enabling goal-directed and proactive responses.
• Figure 3: Overview of streaming KV-caches.(a) Long-term Memory: As the video stream is continuously encoded, key-value (KV) caches from earlier frames are offloaded to CPU memory as long-term memory. (b) Selective Recall: Upon a query, relevant KV-caches are dynamically selected across layers based on attention scores (within $\alpha$ of the max). (c) Short-term Memory: Selected KV-caches are reloaded to GPU and combined with streaming inputs as short-term memory for efficient response generation.
• Figure 4: Comparative radar plots of StreamAgent and existing online video LLMs on diverse benchmarks, including StreamingBench lin2024streamingbench (first three) and OV-Bench huang2025onlinevideounderstandingovbench (last).
• Figure 5: Ablation Study on three core components: (a) The effectiveness of the memory module is evaluated on the medium and long subsets of the VideoMME benchmark. (b) The impact of different planning strategies is evaluated on an OVOBench li2025ovobenchfarvideollmsrealworld subset designed to test temporal foresight. (c) Tool usage is assessed on a separate subset to examine how proactive interaction enhances performance.