StreamingVLM: Real-Time Understanding for Infinite Video Streams

TL;DR

The paper tackles the difficulty of real-time understanding of near-infinite video streams by Vision-Language Models, where full-context attention is too costly and sliding windows fail to maintain coherence. It introduces StreamingVLM, a unified framework that aligns training with streaming inference through a streaming-aware KV cache, contiguous RoPE, and overlapped full-attention training, supported by a data pipeline (Inf-Streams-Train/Eval). Empirically, StreamingVLM achieves real-time commentary up to 8 FPS on a single H100 and attains a 66.18% win rate against GPT-4o mini on Inf-Streams-Eval, while delivering notable improvements on LongVideoBench and OVOBench Realtime without task-specific fine-tuning. The work demonstrates strong potential for practical deployment in real-time streaming assistants and autonomous systems, and provides a new benchmark suite for long-horizon, frame-level alignment in streaming video understanding.

Abstract

Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.

Figures (10)

• Figure 1: Illustration of StreamingVLM vs. existing VLMs. Let $T$ be video length and $W$ the sliding-window size. (a) Full Attention: $O(T^2)$ cost; unbounded memory; degrades beyond training length. (b) Sliding Window (no overlap): bounded memory but short chunks break coherence; long chunks raise latency. (c) Sliding Window (overlap): recomputation per window yields high latency. (d) StreamingVLM (Sliding Window + Reuse KV): reuses states of attention sinks, a short vision window and long text window, preserving history at low latency. “Win rate” is the pairwise win share vs. GPT-4o mini (judge: GPT-5).
• Figure 1: Captioning accuracy (win rate vs. baselines). Baselines with/without chunking fall short; StreamingVLM surpasses strong models such as GPT-4o and produces compelling commentary.(Superscripts for Inf-Streams-Eval: $^{\infty}$ = infinite; † = chunk length 100s. On Livecc-Sports-3K CC, LiveCC has only one mode and cannot be compared against itself, so we show “–”.
• Figure 2: Issues with existing VLMs. (1) Without SFT, models cannot generate cross-round content coherently. (2) With full attention, the context exceeds the training length after processing 2–5 minutes of video and latency becomes prohibitive. (3) With a sliding window, models cannot retain enough context to benefit from efficiency. In contrast, StreamingVLM addresses these issues, enabling coherent commentary, real-time generation, and long-term history.
• Figure 3: Inference scheme of StreamingVLM. We keep 512 attention-sink tokens to stabilize attention, a long text window of 512 recent tokens to preserve long-term memory, and a short vision window covering 16 seconds to track ongoing actions. We use Contiguous RoPE: indices are shifted to stay within a fixed range, keeping positions in-distribution and within the training length.
• Figure 4: Training Strategy. We train with overlapped full attention that mimics test-time attention. (1), (2), (3) and (4) are four training samples, both keeping the attention sinks and overlap later in time.