MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection

TL;DR

MoniTor addresses online video anomaly detection by enabling training-free inference using streaming vision-language models and LLMs guided by anomaly priors and a memory-based scoring scheme. It introduces a dual-memory gating module to capture long- and short-term context, a standard scoring queue for structured anomaly descriptors, and a predictive analysis component to anticipate future events, all integrated into a streaming pipeline. The results on UCF-Crime and XD-Violence show state-of-the-art or competitive performance against training-free and weakly supervised baselines while maintaining real-time processing capabilities. This work highlights the potential of coupling large-scale models with memory-based mechanisms to enable practical, training-free VAD in surveillance contexts.

Abstract

Video Anomaly Detection (VAD) aims to locate unusual activities or behaviors within videos. Recently, offline VAD has garnered substantial research attention, which has been invigorated by the progress in large language models (LLMs) and vision-language models (VLMs), offering the potential for a more nuanced understanding of anomalies. However, online VAD has seldom received attention due to real-time constraints and computational intensity. In this paper, we introduce a novel Memory-based online scoring queue scheme for Training-free VAD (MoniTor), to address the inherent complexities in online VAD. Specifically, MoniTor applies a streaming input to VLMs, leveraging the capabilities of pre-trained large-scale models. To capture temporal dependencies more effectively, we incorporate a novel prediction mechanism inspired by Long Short-Term Memory (LSTM) networks. This ensures the model can effectively model past states and leverage previous predictions to identify anomalous behaviors. Thereby, it better understands the current frame. Moreover, we design a scoring queue and an anomaly prior to dynamically store recent scores and cover all anomalies in the monitoring scenario, providing guidance for LLMs to distinguish between normal and abnormal behaviors over time. We evaluate MoniTor on two large datasets (i.e., UCF-Crime and XD-Violence) containing various surveillance and real-world scenarios. The results demonstrate that MoniTor outperforms state-of-the-art methods and is competitive with weakly supervised methods without training. Code is available at https://github.com/YsTvT/MoniTor.

Figures (7)

• Figure 1: Illustration of MoniTor which detects abnormal events across multiple surveillance perspectives. MoniTor identifies critical security incidents: Abuse, Robbery, Explosion, Fighting and so on.
• Figure 2: The architecture of our MoniTor: (1) Online Vision-Language Model (Sec. \ref{['me:ovlm']}) is used to get each frame's textual summary. (2) Anomaly Priors Integration (Sec. \ref{['me:api']}) is used to serve as a form of "knowledge injection". (3) Dynamic Memory Gating Module (Sec. \ref{['me:dmgm']}) is used to capture historical information while preventing the large model from being misled by historical memory. (4) Behavior Prediction and Dynamic Analysis (Sec. \ref{['me::bpada']}) leverages frame-to-frame predictive cues to facilitate robust anomaly detection through comparative analysis of temporal discrepancies. (5) Standard Scoring Queue (Sec. \ref{['me:ssq']}) is used to guide the large model on how to identify and understand anomalies. (6) Score Optimization and Weight Assignment (Sec. \ref{['me:wa']}) adjusts LLMs' scoring results based on different context, better aiding LLMs in distinguishing abnormal behaviors.
• Figure 3: We present qualitative results of our MoniTor on test videos. For each video, we graph the anomaly scores across the frames by our approach. Alongside this, we show keyframes with their corresponding temporal summaries, in which blue bounding boxes denote normal frames and red for those deemed anomalous—thus showcasing the correlation between the anomaly scores, the visual content, and their descriptions. Notably, the ground-truth anomalies are highlighted.
• Figure 4: We present real-world tests using MoniTor. For each video, we graph the anomaly scores across the frames by our approach. Alongside this, we show keyframes, in which blue bounding boxes denote normal frames and red for those deemed anomalous—thus showcasing the correlation between the anomaly scores and the visual content. Notably, the ground-truth anomalies are highlighted.
• Figure 5: Results of MoniTor on UCF-Crime over $\alpha$ used for Weight Assignment.