StreamingTOM: Streaming Token Compression for Efficient Video Understanding

TL;DR

StreamingTOM addresses the dual bottlenecks of streaming video understanding under strict causality: heavy pre-LLM token processing and unbounded post-LLM kv-cache growth. It introduces CTR, a strictly causal pre-LLM token reduction that enforces a fixed per-frame budget $G$, reducing per-frame tokens from $N$ to $G$ and lowering prefill complexity from $O(TNLd^{2})$ to $O(TGLd^{2})$. It complements CTR with OQM, which stores retained groups in 4-bit precision and retrieves only the most relevant groups on demand, bounding active memory to $O(kGd)$ while achieving total storage $O(TGd)/4$. The two-stage framework, $\text{StreamingTOM} = \mathrm{OQM}_{16\to4} \circ \mathrm{CTR}_{N\to G}$, delivers $15.7\times$ kv-cache compression, $1.2\times$ lower peak memory, and $2\times$ faster TTFT compared with prior training-free SOTA, while maintaining strong accuracy on offline benchmarks ($63.8\%$ average) and RVS ($55.8\%/3.7$). By ensuring bounded memory growth over long horizons (e.g., reducing one-hour kv-cache from $18.8$ GB to $1.2$ GB on LLaVA-OV-7B), StreamingTOM enables practical, real-time streaming video understanding without retraining.

Abstract

Unlike offline processing, streaming video vision-language models face two fundamental constraints: causality and accumulation. Causality prevents access to future frames that offline methods exploit, while accumulation causes tokens to grow unbounded, creating efficiency bottlenecks. However, existing approaches only regulate post-LLM kv-cache, leaving costly pre-LLM prefill unchanged. We introduce StreamingTOM, a training-free, plug-and-play two-stage framework that addresses both pre-LLM and post-LLM bottlenecks with predictable latency. Causal Temporal Reduction imposes a fixed per-frame budget and selects tokens based on adjacent-frame changes and token saliency, drastically reducing per-frame prefill cost by processing only a compact subset of visual tokens per frame instead of all visual tokens. Online Quantized Memory stores tokens in 4-bit format, retrieves relevant groups on demand, and dequantizes them, keeping the active kv-cache bounded regardless of stream length. Experiments demonstrate our method achieves $15.7\times$ kv-cache compression, $1.2\times$ lower peak memory and $2\times$ faster TTFT compared to prior SOTA. StreamingTOM maintains state-of-the-art accuracy among training-free methods with an average of $63.8\%$ on offline benchmarks and $55.8\%/3.7$ on RVS. These results highlight the practical benefits of our two-stage approach for efficient streaming video understanding with bounded growth.

Figures (6)

• Figure 1: Left: StreamingTOM (streamingtoken compression) is a training-free, two-stage framework for efficient streaming video understanding: Causal Temporal Reduction selects pre-LLM tokens with a causal, fixed-budget policy, and Online Quantized Memory maintains compact kv-cache via 4-bit quantization with on-demand retrieval. Right: on LLaVA-OV-7B, StreamingTOM achieves a $15.7\times$ kv-cache compression ratio; compared to prior SOTA (LiveVLM), it delivers $1.2\times$ lower peak memory and $2\times$ faster TTFT.
• Figure 2: Overview of the StreamingTOM architecture. The framework consists of two coordinated pipelines. The vision pipeline encodes each frame and applies Causal Temporal Reduction to condense redundant tokens into compact groups, which are written to an online memory for reuse. The query pipeline processes questions and drives the decoder to interact with the memory through Online Quantized Memory, which stores groups at 4-bit precision, retrieves at most $k$ groups on demand, and dequantizes them for efficient generation.
• Figure 3: Overview of the CTR compression pipeline. The algorithm processes visual tokens from consecutive frames using a 2-frame window (current and previous), producing a binary classification (static, dynamic) through similarity comparison. The adaptive budget allocation dynamically distributes compression resources based on content, followed by dual-path processing: dpc clustering for static tokens and attention-based selection for dynamic tokens.
• Figure 4: Two qualitative examples from RVS Movie requiring long-horizon reasoning. StreamingTOM provides faithful answers consistent with ground truth, accurately capturing fine-grained semantics such as "subway station" and multi-party interactions. These results illustrate the model's ability to maintain causal reasoning and long-horizon consistency in real streaming scenarios.
• Figure 5: Complete processing timeline comparison. The top row shows the baseline, while the bottom row shows StreamingTOM. For a 64-frame stream at batch size 8 with 50 tokens per frame, StreamingTOM incurs minimal overhead: OQM introduces 7.3ms for kv storage, 6.9ms for retrieval, and 28.7ms for 4-bit reconstruction. In contrast, CTR delivers 3.6$\times$ prefill acceleration from 337.8ms to 92.8ms, yielding an efficient 0.20s query TTFT and demonstrating substantial performance gains at negligible cost.