VQPy: An Object-Oriented Approach to Modern Video Analytics
Shan Yu, Zhenting Zhu, Yu Chen, Hanchen Xu, Pengzhan Zhao, Yang Wang, Arthi Padmanabhan, Hugo Latapie, Harry Xu
TL;DR
VQPy addresses the challenge of expressing and optimizing complex video queries by adopting an object-centric paradigm that centers on video objects and their interactions. It provides a Python-based frontend (VObj, Relation, Query) and an object-centric backend with a DAG planner, execution engine, and an extensible optimization framework, enabling efficient plan generation, object-level computation reuse, and DAG optimizations. Empirically, VQPy delivers substantial speedups (up to 12.6x vs handcrafted pipelines and up to 15.2x vs EVA) across multiple surveillance datasets, while achieving competitive or superior accuracy compared to multimodal LLMs, and it demonstrates real-world adoption within Cisco's DeepVision. The work highlights practical impact through open-sourcing and a pathway for integrating specialized models, frame filters, and higher-order queries in a flexible, extensible system for modern video analytics.
Abstract
Video analytics is widely used in contemporary systems and services. At the forefront of video analytics are video queries that users develop to find objects of particular interest. Building upon the insight that video objects (e.g., human, animals, cars, etc.), the center of video analytics, are similar in spirit to objects modeled by traditional object-oriented languages, we propose to develop an object-oriented approach to video analytics. This approach, named VQPy, consists of a frontend$\unicode{x2015}$a Python variant with constructs that make it easy for users to express video objects and their interactions$\unicode{x2015}$as well as an extensible backend that can automatically construct and optimize pipelines based on video objects. We have implemented and open-sourced VQPy, which has been productized in Cisco as part of its DeepVision framework.