StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression

TL;DR

StreamKV addresses the challenges of streaming video question answering with long videos by introducing semantic segment partitioning and a summary vector to preserve segment-level information. It unifies KV cache retrieval and compression in a training-free framework guided by a semantic guidance prompt and a layer-adaptive budget allocation to maximize informative content. On StreamingBench, StreamKV achieves higher accuracy than existing online Video-LLMs while reducing memory usage and latency, including strong performance under substantial KV compression. The approach enables robust, memory-efficient streaming Video-LLMs suitable for real-time, real-world applications.

Abstract

Video Large Language Models (Video-LLMs) have demonstrated significant potential in the areas of video captioning, search, and summarization. However, current Video-LLMs still face challenges with long real-world videos. Recent methods have introduced a retrieval mechanism that retrieves query-relevant KV caches for question answering, enhancing the efficiency and accuracy of long real-world videos. However, the compression and retrieval of KV caches are still not fully explored. In this paper, we propose \textbf{StreamKV}, a training-free framework that seamlessly equips Video-LLMs with advanced KV cache retrieval and compression. Compared to previous methods that used uniform partitioning, StreamKV dynamically partitions video streams into semantic segments, which better preserves semantic information. For KV cache retrieval, StreamKV calculates a summary vector for each segment to retain segment-level information essential for retrieval. For KV cache compression, StreamKV introduces a guidance prompt designed to capture the key semantic elements within each segment, ensuring only the most informative KV caches are retained for answering questions. Moreover, StreamKV unifies KV cache retrieval and compression within a single module, performing both in a layer-adaptive manner, thereby further improving the effectiveness of streaming video question answering. Extensive experiments on public StreamingVQA benchmarks demonstrate that StreamKV significantly outperforms existing Online Video-LLMs, achieving superior accuracy while substantially improving both memory efficiency and computational latency. The code has been released at https://github.com/sou1p0wer/StreamKV.

Figures (4)

• Figure 1: Comparison with ReKV on StreamingBench in terms of VideoQA accuracy, memory usage, and latency
• Figure 2: Video Segment Processing
• Figure 3: StreamKV workflow. StreamKV dynamically partitions video streams into semantic segments, and calculates a summary vector for each segment. These segments are sequentially encoded to generate frame-level KV blocks. KV compression is applied to all blocks except for those generated by the summary vector. We store the compressed KV blocks and summary KV block in the KV Bank. Upon receiving a question, StreamKV retrieves query-relevant KV blocks from the KV Bank to generate responses. Both the compression and retrieval are performed using our proposed unified layer-adaptive selection module.
• Figure 4: Accuracy versus Retrieved Frames Comparison for StreamKV and ReKV.