Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries

TL;DR

Vista tackles streaming video QA under post-hoc queries by decoupling memory and computation through scene-aware optimization. It introduces scene-aware segmentation, compression, and recall to maintain long-context reasoning with bounded GPU memory, and remains model-agnostic for seamless integration with vision-language backbones. The method uses online scene boundaries, temporal-spatial compression into compact scene tokens, and a top-$k$ recall mechanism guided by query relevance, enabling on-demand retrieval of full-resolution frames when needed. Empirically, Vista delivers state-of-the-art performance on StreamingBench while showing strong offline long-form results on MLVU and EgoSchema, with notably lower memory usage and latency compared with prior streaming models. Such capabilities offer practical, scalable real-time video understanding for real-world streaming applications.

Abstract

Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.

Figures (6)

• Figure 1: Illustration of the streaming video QA setting with post-hoc queries and scene-level attention. User questions may arrive at any time during video playback, and the model's attention distribution typically concentrates on a single coherent scene.
• Figure 2: The framework of Vista. (a) Scene-aware segmentation. (b) Scene-aware compression. (c) Scene-aware recall.
• Figure 3: Performance comparison with increasing input frames across different strategies: (a) memory usage; (b) inference latency.
• Figure 4: Offline video question-answering on different long-form benchmarks. Most of the data is cited from Dispider qian2025dispider.
• Figure 5: Sensitivity analysis of four different hyperparameters for streaming video QA: (a) Window Size $a$; (b) Similarity threshold $\tau$; (c) Compression-recovery ratio $m / k$; (d) Overlap step size.