Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting

TL;DR

The paper tackles the challenge of open-world video anomaly detection using frozen vision-language models by showing that abstract prompts underutilize VLM reasoning. It introduces ASK-HINT, a training-free framework with three components: per-class fine-grained prompts, semantic compression into a compact guiding set $ ext{Q}^*$, and a structured two-stage inference that yields both a normal/abnormal decision and an interpretable justification. Empirical results on UCF-Crimes and XD-Violence demonstrate state-of-the-art $AUC$ for training-free VAD (e.g., $89.83\%$ on UCF-Crime and $90.31\%$ on XD-Violence), along with strong cross-dataset and cross-class generalization and transparent reasoning traces. The findings underscore the importance of prompt granularity and semantic structure for grounding VLM reasoning in concrete human–object interactions, offering a practical, deployment-friendly alternative to costly fine-tuning. Overall, ASK-HINT advances explainable, zero-shot VAD with robust generalization, while highlighting future work in temporal reasoning and dynamic prompting.

Abstract

Prompting has emerged as a practical way to adapt frozen vision-language models (VLMs) for video anomaly detection (VAD). Yet, existing prompts are often overly abstract, overlooking the fine-grained human-object interactions or action semantics that define complex anomalies in surveillance videos. We propose ASK-Hint, a structured prompting framework that leverages action-centric knowledge to elicit more accurate and interpretable reasoning from frozen VLMs. Our approach organizes prompts into semantically coherent groups (e.g. violence, property crimes, public safety) and formulates fine-grained guiding questions that align model predictions with discriminative visual cues. Extensive experiments on UCF-Crime and XD-Violence show that ASK-Hint consistently improves AUC over prior baselines, achieving state-of-the-art performance compared to both fine-tuned and training-free methods. Beyond accuracy, our framework provides interpretable reasoning traces towards anomaly and demonstrates strong generalization across datasets and VLM backbones. These results highlight the critical role of prompt granularity and establish ASK-Hint as a new training-free and generalizable solution for explainable video anomaly detection.

Figures (8)

• Figure 1: Performance of video anomaly detection w.r.t. prompt granularity. Given the same video input, an abstract prompt leads to a false prediction, while fine-grained action prompts (e.g. “punching”, “attacking”) elicit the correct abnormal classification from the model.
• Figure 2: Comparison between coarse and fine-grained prompts across crime categories in the UCF-Crime dataset sultani2018real, where fine-grained prompts significantly improve AUC over corse prompts.
• Figure 3: Prompt similarity heatmap across anomaly classes. Cosine similarities between average prompt embeddings reveal semantically coherent clusters, such as Arson–Explosion and Stealing–Shoplifting–Robbery, supporting the hypothesis that anomaly categories share fine-grained action patterns.
• Figure 4: Overall pipeline of ASK-HINT. Step 1: class-wise fine-grained action questions are generated for each anomaly category (Action Generation). Step 2: questions that reflect the same underlying action primitives are grouped together (Clustering),which we mark with the same color. Step 3: each cluster is condensed into representative guiding questions, yielding a compact and transferable prompt set $\mathcal{Q}^*$ (Summarizing).
• Figure 5: Video anomaly detection with the proposed ASK-HINT, using the UCF-Crime sultani2018real dataset as an example. It guides the VLM in two stages: (1) binary decision between normal and abnormal events, and (2) group-level classification with justification.