VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models

TL;DR

VERA addresses explainable video anomaly detection by enabling frozen vision-language models to reason about anomalies through a learned set of guiding questions optimized via verbalized learning between a learner and an optimizer, using only coarse video-level labels. The learned questions are embedded in prompts to compute segment-level anomaly scores, which are refined to frame-level scores through scene-context ensemble and temporal smoothing, with explanations generated from the reasoning process. Empirical results on UCF-Crime and XD-Violence show that VERA achieves state-of-the-art explainable VAD performance among methods that do not fine-tune backbones, while maintaining low training overhead and strong generalization across models and datasets. This work demonstrates that verbalized learning can effectively adapt frozen VLMs to temporally structured tasks like VAD, enabling accurate detection and human-interpretable explanations without costly IT or module additions, with potential impact on scalable anomaly surveillance applications.

Abstract

The rapid advancement of vision-language models (VLMs) has established a new paradigm in video anomaly detection (VAD): leveraging VLMs to simultaneously detect anomalies and provide comprehendible explanations for the decisions. Existing work in this direction often assumes the complex reasoning required for VAD exceeds the capabilities of pretrained VLMs. Consequently, these approaches either incorporate specialized reasoning modules during inference or rely on instruction tuning datasets through additional training to adapt VLMs for VAD. However, such strategies often incur substantial computational costs or data annotation overhead. To address these challenges in explainable VAD, we introduce a verbalized learning framework named VERA that enables VLMs to perform VAD without model parameter modifications. Specifically, VERA automatically decomposes the complex reasoning required for VAD into reflections on simpler, more focused guiding questions capturing distinct abnormal patterns. It treats these reflective questions as learnable parameters and optimizes them through data-driven verbal interactions between learner and optimizer VLMs, using coarsely labeled training data. During inference, VERA embeds the learned questions into model prompts to guide VLMs in generating segment-level anomaly scores, which are then refined into frame-level scores via the fusion of scene and temporal contexts. Experimental results on challenging benchmarks demonstrate that the learned questions of VERA are highly adaptable, significantly improving both detection performance and explainability of VLMs for VAD.

Figures (13)

• Figure 1: VERA renders frozen VLMs to describe and reason with learnable guiding questions learned from coarsely labeled data.
• Figure 2: The overall training pipeline in VERA aims to optimize VAD guiding questions iteratively. In each iteration $t$, the optimization is verbalized by providing verbal instructions for the learner and optimizer to follow. They will generate predictions and new guiding questions, respectively.
• Figure 3: VERA computes anomaly scores with $\mathbf{Q}^{*}$ in three steps.
• Figure 4: Effect of the number of guiding questions on AUC.
• Figure 5: Given $\mathbf{Q}^{*}$ by VERA, the frozen VLM (InternVL2-8B) will reason and explain the scene based on it. For illustration, we take as an example the video "Arrest007_x264" from UCF-Crime and include 6 scenes here. The complete anomaly scores are shown in Fig. \ref{['fig:case']}.