Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

TL;DR

This work tackles the challenge of real-time, long-sequence streaming video understanding with multi-turn interactions. It introduces StreamChat, a training-free framework that leverages a hierarchical memory system ($M_l$, $M_s$, $M_d$) and a three-thread system scheduler to compress video content and support rapid, multi-round reasoning. A new StreamBench benchmark is proposed to evaluate online streaming performance across diverse video types and six task types with latency metrics. Empirical results show StreamChat achieves real-time processing at 32 FPS with sub-0.9s latency and outperforms prior methods in both online and offline benchmarks, validated by extensive ablations. The work advances practical streaming video understanding with robust memory management and retrieval-driven reasoning, enabling more responsive and context-aware video-language interactions.

Abstract

Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. $\StreamChat$ leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. Extensive evaluations on StreamBench and other public benchmarks demonstrate that StreamChat significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat: https://github.com/hmxiong/StreamChat.

Figures (13)

• Figure 1: Performance comparison between $\texttt{\normalsize S\footnotesize TREAM\normalsize C\footnotesize HAT}$ and previous Video-LLMs.
• Figure 2: The comparisons between StreamChat and other methods (§\ref{['introduction']}). Offline methods process entire videos, leading to information loss and limited to a single interaction. Previous online methods chen2024videollmzhang2024flash enable multi-round interactions but still suffer from slow processing and answer correctly. The proposed method achieves real-time video processing, improving the efficiency and accuracy with memory support.
• Figure 3: Benchmark overview (§\ref{['benchmark']}). Our benchmark covers 4 key domains and 16 sub-class video types. These videos exhibit a broader distribution of length, with 6 different types that are evenly distributed.
• Figure 4: Overview of StreamChat (§\ref{['streamchat']}), which comprises three main components: (i) Selective frame stacking, which prepares vision features for processing, including encoding frames and filling the vision buffer; (ii) Memory formation, where vision features are organized into structured memory; (iii) Contextual summarization, utilizing hierarchical memory to respond to user queries by providing relevant context.
• Figure 5: The hierarchical memory storage (§\ref{['hierarchical']}). (a) Long-short term memory, where the long memory tree $M_l$ and short-term memory $M_s$ are constructed along the video time line. (b) The dialogue memory $M_d$ is updated after each inference conversation for managing the dialogue histories.