Video Anomaly Detection with Structured Keywords

TL;DR

This work tackles real-time, interpretable video anomaly detection by reframing features as text-based keywords. It introduces a two-stage induction-deduction pipeline that uses frame descriptions from open-source foundational models to compute a weight vector over the top $k$ keywords via TF-IDF, then encodes frames into keyword representations for a binary classifier. The approach achieves comparable ROC AUC on UCSD Ped2 ($0.865$), ShanghaiTech ($0.745$), and CUHK Avenue ($0.742$) while sacrificing some accuracy on more complex scenes in exchange for near real-time, edge-friendly performance and clear interpretability. By emphasizing interpretability and low-resource inference, the method offers a practical alternative for surveillance applications with limited computing capabilities, paving the way for broader adoption of text-based feature representations in video anomaly detection.

Abstract

This paper focuses on detecting anomalies in surveillance video using keywords by leveraging foundational models' feature representation generalization capabilities. We present a novel, lightweight pipeline for anomaly classification using keyword weights. Our pipeline employs a two-stage process: induction followed by deduction. In induction, descriptions are generated from normal and anomalous frames to identify and assign weights to relevant keywords. In deduction, inference frame descriptions are converted into keyword encodings using induction-derived weights for input into our neural network for anomaly classification. We achieved comparable performance on the three benchmarks UCSD Ped2, Shanghai Tech, and CUHK Avenue, with ROC AUC scores of 0.865, 0.745, and 0.742, respectively. These results are achieved without temporal context, making such a system viable for real-time applications. Our model improves implementation setup, interpretability, and inference speed for surveillance devices on the edge, introducing a performance trade-off against other video anomaly detection systems. As the generalization capabilities of open-source foundational models improve, our model demonstrates that the exclusive use of text for feature representations is a promising direction for efficient real-time interpretable video anomaly detection.

Figures (6)

• Figure 1: Pipeline Overview. FM abbreviates the foundational models necessary for generating text descriptions from frame input. TF-IDF abbreviates the Term Frequency-Inverse Document Frequency score, which we use to weigh keywords.
• Figure 2: Frame description generation. $F_{norm}$ represents labeled normal frames with their respective descriptions $D_{norm}$ and $F_{anom}$ represents labeled anomalous frames with their respective descriptions $D_{anom}$ generated by the pre-trained foundational model.
• Figure 3: Creating the keyword encoding. Description $D_{deduct}$ is generated by passing frame $F_{deduct}$ into the pre-trained foundational model. Then the description is mapped into a keyword encoding $E_{deduct}$ using the keyword weights $w_{keywords}$ from the induction stage.
• Figure 4: Feed-forward network for binary classification. $FC-n$ stands for the fully-connected layer number $n$. The input and output dimensions are represented by $(i, o)$, with $i$ for input and $o$ for output. The dimension size $k$ is based on the number of keywords generated from induction. The probability $P(A)$ represents the chance that the inputted keyword encoding is an anomaly.
• Figure 5: VAD benchmark examples. These include UCSD Ped2 mahadevan2010anomaly (left), ShanghaiTech liu2018future (center), and the CUHK Avenue lu2013abnormal (right) datasets. Images in the top row depict normal occurrences, and the bottom row depicts anomaly occurrences.