Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding

TL;DR

This work addresses the deployment bottlenecks of VLM-based online video understanding by designing Venus, an edge–cloud disaggregated system that performs memory construction and keyframe retrieval on the edge while delegating reasoning to the cloud. It introduces scene segmentation and frame clustering to build a sparse, hierarchical memory, and proposes a threshold-driven progressive sampling algorithm for diverse, query-relevant keyframe retrieval. Venus demonstrates up to 15×–131× reductions in end-to-end latency with comparable or superior reasoning accuracy across benchmarks, edge devices, and cloud models. The approach offers practical pathways to real-time, memory-grounded video understanding in edge environments with scalable retrieval and reduced cloud transmission.

Abstract

Vision-language models (VLMs) have demonstrated impressive multimodal comprehension capabilities and are being deployed in an increasing number of online video understanding applications. While recent efforts extensively explore advancing VLMs' reasoning power in these cases, deployment constraints are overlooked, leading to overwhelming system overhead in real-world deployments. To address that, we propose Venus, an on-device memory-and-retrieval system for efficient online video understanding. Venus proposes an edge-cloud disaggregated architecture that sinks memory construction and keyframe retrieval from cloud to edge, operating in two stages. In the ingestion stage, Venus continuously processes streaming edge videos via scene segmentation and clustering, where the selected keyframes are embedded with a multimodal embedding model to build a hierarchical memory for efficient storage and retrieval. In the querying stage, Venus indexes incoming queries from memory, and employs a threshold-based progressive sampling algorithm for keyframe selection that enhances diversity and adaptively balances system cost and reasoning accuracy. Our extensive evaluation shows that Venus achieves a 15x-131x speedup in total response latency compared to state-of-the-art methods, enabling real-time responses within seconds while maintaining comparable or even superior reasoning accuracy.

Figures (12)

• Figure 1: An online video understanding application empowered by VLMs.
• Figure 2: Latency breakdown for video understanding task, including communication, cloud, and on-device processing. Measured on NVIDIA AGX Orin using an 8 FPS EgoSchema video. VLMs run on a server with one NVIDIA L40S GPU. Video-RAG, BOLT, and AKS sample 32 frames.
• Figure 3: An edge-cloud disaggregated architecture.
• Figure 4: Embedding latency versus FPS across edge devices, with the threshold marking the maximum FPS each device can sustain for real-time embedding.
• Figure 5: (a) Accuracy on the short split of Video-MME (w/o subtitles). Adding redundant frames to the vector database leads to performance degradation, highlighting the importance of diversity in frame selection. (b) Visualization of frame selection using Top-K retrieval. The blue curve shows similarity scores across 256 uniformly sampled frames; red dots indicate selected frames, which are concentrated around adjacent timestamps. (c) Retrieved frames using Top-K selection. All selected frames focus exclusively on the “cucumber” segment, ignoring other answer options and resulting in incorrect predictions.