SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models

TL;DR

SurveillanceVQA-589K tackles the need for deep, semantics-rich understanding of surveillance video by introducing a large-scale, open-ended video QA dataset spanning normal and abnormal events. The authors present a hybrid annotation pipeline that fuses human captions with LVLM-generated QA content and a multi-dimensional evaluation protocol to assess contextual, temporal, and causal understanding. They benchmark eight LVLMs and reveal systematic gaps in causal inference and anomaly reasoning, underscoring current limitations for safety-critical surveillance tasks. The dataset and findings provide a practical resource for advancing video-language understanding in intelligent monitoring and incident-response systems, and point to the need for stronger temporal-causal reasoning and domain-specific fine-tuning.

Abstract

Understanding surveillance video content remains a critical yet underexplored challenge in vision-language research, particularly due to its real-world complexity, irregular event dynamics, and safety-critical implications. In this work, we introduce SurveillanceVQA-589K, the largest open-ended video question answering benchmark tailored to the surveillance domain. The dataset comprises 589,380 QA pairs spanning 12 cognitively diverse question types, including temporal reasoning, causal inference, spatial understanding, and anomaly interpretation, across both normal and abnormal video scenarios. To construct the benchmark at scale, we design a hybrid annotation pipeline that combines temporally aligned human-written captions with Large Vision-Language Model-assisted QA generation using prompt-based techniques. We also propose a multi-dimensional evaluation protocol to assess contextual, temporal, and causal comprehension. We evaluate eight LVLMs under this framework, revealing significant performance gaps, especially in causal and anomaly-related tasks, underscoring the limitations of current models in real-world surveillance contexts. Our benchmark provides a practical and comprehensive resource for advancing video-language understanding in safety-critical applications such as intelligent monitoring, incident analysis, and autonomous decision-making.

Figures (7)

• Figure 1: Proposition of our SurveillanceVQA tasks.
• Figure 2: Our overall framework, including QA generation and evaluation.
• Figure 3: Procedure of integrating annotations.
• Figure 4: Distribution of event durations on the training/test sets
• Figure 5: Comparisons of scores from different LVLMs across different QA tasks. Left: normal video clips vs. normal QA tasks. Middle: abnormal video clips vs. normal QA tasks. Right: abnormal video clips vs. abnormal QA tasks.