Evaluation of Vision-LLMs in Surveillance Video

TL;DR

Zero-shot anomaly recognition in surveillance video is addressed by framing detection as language-grounded reasoning using small vision-language models to produce textual descriptions of clips, followed by a frozen natural language inference score to select anomaly labels. The authors implement a training-free pipeline and systematically evaluate four open vision-LLMs on UCF-Crime and RWF-2000 under varied prompting and privacy-preserving conditions, reporting macro-Top-1 accuracy and FP metrics. Key findings show few-shot prompts can boost accuracy for some models but may raise false positives, while privacy filters modestly reduce accuracy and, in the case of full-body GANs, cause notable inconsistencies; the work maps current capabilities to simple versus complex spatial cues and outlines concrete improvements for spatial grounding. The study demonstrates the potential of language-grounded, training-free pipelines for real-world video understanding and provides actionable directions—such as structure-aware prompts, temporal memory, and priors—to enhance robustness in surveillance contexts.

Abstract

The widespread use of cameras in our society has created an overwhelming amount of video data, far exceeding the capacity for human monitoring. This presents a critical challenge for public safety and security, as the timely detection of anomalous or criminal events is crucial for effective response and prevention. The ability for an embodied agent to recognize unexpected events is fundamentally tied to its capacity for spatial reasoning. This paper investigates the spatial reasoning of vision-language models (VLMs) by framing anomalous action recognition as a zero-shot, language-grounded task, addressing the embodied perception challenge of interpreting dynamic 3D scenes from sparse 2D video. Specifically, we investigate whether small, pre-trained vision--LLMs can act as spatially-grounded, zero-shot anomaly detectors by converting video into text descriptions and scoring labels via textual entailment. We evaluate four open models on UCF-Crime and RWF-2000 under prompting and privacy-preserving conditions. Few-shot exemplars can improve accuracy for some models, but may increase false positives, and privacy filters -- especially full-body GAN transforms -- introduce inconsistencies that degrade accuracy. These results chart where current vision--LLMs succeed (simple, spatially salient events) and where they falter (noisy spatial cues, identity obfuscation). Looking forward, we outline concrete paths to strengthen spatial grounding without task-specific training: structure-aware prompts, lightweight spatial memory across clips, scene-graph or 3D-pose priors during description, and privacy methods that preserve action-relevant geometry. This positions zero-shot, language-grounded pipelines as adaptable building blocks for embodied, real-world video understanding. Our implementation for evaluating VLMs is publicly available at: https://github.com/pascalbenschopTU/VLLM_AnomalyRecognition

Figures (1)

• Figure 1: Few-shot prompting impact on models accuracy level.