Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-Centric Anomaly Detection in Surveillance Videos Using YOLO-World and Spatio-Temporal Deep Learning

Mohammad Ali Etemadi Naeen, Hoda Mohammadzade, Saeed Bagheri Shouraki

TL;DR

The paper tackles automated anomaly detection in surveillance videos, focusing on overcoming background clutter, scene variability, and class imbalance. It introduces a human-centric preprocessing pipeline that uses open-vocabulary detection (YOLO-World) and ByteTrack tracking to produce foreground-focused frames, which are then analyzed by an InceptionV3-based spatial feature extractor and a BiLSTM temporal model for multi-class classification. Empirically, the approach achieves a mean accuracy of 92.41% across three random splits on a five-class UCF-Crime subset, with per-class F1-scores ≥ 0.85 and ROC-AUC values above 0.98, outperforming several baselines. The results suggest foreground-focused preprocessing substantially improves anomaly discrimination and generalization, offering a practical, efficient pathway for real-world surveillance analytics.

Abstract

Anomaly detection in surveillance videos remains a challenging task due to the diversity of abnormal events, class imbalance, and scene-dependent visual clutter. To address these issues, we propose a robust deep learning framework that integrates human-centric preprocessing with spatio-temporal modeling for multi-class anomaly classification. Our pipeline begins by applying YOLO-World - an open-vocabulary vision-language detector - to identify human instances in raw video clips, followed by ByteTrack for consistent identity-aware tracking. Background regions outside detected bounding boxes are suppressed via Gaussian blurring, effectively reducing scene-specific distractions and focusing the model on behaviorally relevant foreground content. The refined frames are then processed by an ImageNet-pretrained InceptionV3 network for spatial feature extraction, and temporal dynamics are captured using a bidirectional LSTM (BiLSTM) for sequence-level classification. Evaluated on a five-class subset of the UCF-Crime dataset (Normal, Burglary, Fighting, Arson, Explosion), our method achieves a mean test accuracy of 92.41% across three independent trials, with per-class F1-scores consistently exceeding 0.85. Comprehensive evaluation metrics - including confusion matrices, ROC curves, and macro/weighted averages - demonstrate strong generalization and resilience to class imbalance. The results confirm that foreground-focused preprocessing significantly enhances anomaly discrimination in real-world surveillance scenarios.

Human-Centric Anomaly Detection in Surveillance Videos Using YOLO-World and Spatio-Temporal Deep Learning

TL;DR

The paper tackles automated anomaly detection in surveillance videos, focusing on overcoming background clutter, scene variability, and class imbalance. It introduces a human-centric preprocessing pipeline that uses open-vocabulary detection (YOLO-World) and ByteTrack tracking to produce foreground-focused frames, which are then analyzed by an InceptionV3-based spatial feature extractor and a BiLSTM temporal model for multi-class classification. Empirically, the approach achieves a mean accuracy of 92.41% across three random splits on a five-class UCF-Crime subset, with per-class F1-scores ≥ 0.85 and ROC-AUC values above 0.98, outperforming several baselines. The results suggest foreground-focused preprocessing substantially improves anomaly discrimination and generalization, offering a practical, efficient pathway for real-world surveillance analytics.

Abstract

Anomaly detection in surveillance videos remains a challenging task due to the diversity of abnormal events, class imbalance, and scene-dependent visual clutter. To address these issues, we propose a robust deep learning framework that integrates human-centric preprocessing with spatio-temporal modeling for multi-class anomaly classification. Our pipeline begins by applying YOLO-World - an open-vocabulary vision-language detector - to identify human instances in raw video clips, followed by ByteTrack for consistent identity-aware tracking. Background regions outside detected bounding boxes are suppressed via Gaussian blurring, effectively reducing scene-specific distractions and focusing the model on behaviorally relevant foreground content. The refined frames are then processed by an ImageNet-pretrained InceptionV3 network for spatial feature extraction, and temporal dynamics are captured using a bidirectional LSTM (BiLSTM) for sequence-level classification. Evaluated on a five-class subset of the UCF-Crime dataset (Normal, Burglary, Fighting, Arson, Explosion), our method achieves a mean test accuracy of 92.41% across three independent trials, with per-class F1-scores consistently exceeding 0.85. Comprehensive evaluation metrics - including confusion matrices, ROC curves, and macro/weighted averages - demonstrate strong generalization and resilience to class imbalance. The results confirm that foreground-focused preprocessing significantly enhances anomaly discrimination in real-world surveillance scenarios.
Paper Structure (20 sections, 7 equations, 3 figures, 3 tables)

This paper contains 20 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Categorization of Anomaly Detection Approaches
  4. Deep Learning Architectures for Spatiotemporal Feature Extraction
  5. CNNs and RNNs
  6. Combined and Spatiotemporal Models (STNs)
  7. Advanced Deep Learning Mechanisms
  8. Proposed Methodology
  9. Human-Centric Region Extraction via Open-Vocabulary Detection and Temporal Tracking
  10. Video Trimming and Frame Sampling
  11. Frame-Level Feature Extraction using Deep CNN Backbone
  12. Temporal Feature Aggregation
  13. Anomaly Classification
  14. EXPERIMENT AND RESULTS
  15. Dataset
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Conceptual overview of the proposed anomaly detection pipeline. The process begins with raw video input, proceeds through human-centric preprocessing (detection, tracking, and background blurring), and concludes with spatio-temporal classification using the InceptionV3 + BiLSTM architecture.
  • Figure 2: Flowchart of the proposed anomaly detection pipeline. Input video clips undergo human-centric preprocessing using YOLO-World detection and background blurring to suppress scene-specific distractions. A fixed sequence of 32 frames is sampled and fed into an InceptionV3 backbone for spatial feature extraction. Temporal dynamics are modeled via a bidirectional LSTM (BiLSTM), followed by fully connected layers for multi-class anomaly classification.
  • Figure 3: Model Performance Metrics. (a) illustrates the training and validation loss/accuracy over epochs, showing effective convergence. (b) displays the confusion matrix, demonstrating minimal cross-class errors. (c) presents the multi-class ROC curves, highlighting the strong discriminative power with high AUC values.