Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions

Pratik K. Mishra, Alex Mihailidis, Shehroz S. Khan

TL;DR

This paper addresses privacy concerns in video anomaly detection by advocating skeleton-based representations that obviate facial identity and appearance details. It surveys deep learning approaches organized into four learning paradigms—reconstruction, prediction, their combinations, and other methods—across 29 papers, highlighting graph-based and sequence-based models that operate on spatio-temporal skeleton graphs. The review identifies datasets, evaluation metrics, and practical challenges such as pose estimation accuracy, multi-person tracking, and thresholding, and discusses future directions like privacy-preserving multimodal fusion and federated learning. The work aims to catalyze the adoption of privacy-protecting anomaly detection in homes, care facilities, and other sensitive environments, ultimately improving safety and quality of life while maintaining individual privacy.

Abstract

The existing methods for video anomaly detection mostly utilize videos containing identifiable facial and appearance-based features. The use of videos with identifiable faces raises privacy concerns, especially when used in a hospital or community-based setting. Appearance-based features can also be sensitive to pixel-based noise, straining the anomaly detection methods to model the changes in the background and making it difficult to focus on the actions of humans in the foreground. Structural information in the form of skeletons describing the human motion in the videos is privacy-protecting and can overcome some of the problems posed by appearance-based features. In this paper, we present a survey of privacy-protecting deep learning anomaly detection methods using skeletons extracted from videos. We present a novel taxonomy of algorithms based on the various learning approaches. We conclude that skeleton-based approaches for anomaly detection can be a plausible privacy-protecting alternative for video anomaly detection. Lastly, we identify major open research questions and provide guidelines to address them.

Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions

TL;DR

This paper addresses privacy concerns in video anomaly detection by advocating skeleton-based representations that obviate facial identity and appearance details. It surveys deep learning approaches organized into four learning paradigms—reconstruction, prediction, their combinations, and other methods—across 29 papers, highlighting graph-based and sequence-based models that operate on spatio-temporal skeleton graphs. The review identifies datasets, evaluation metrics, and practical challenges such as pose estimation accuracy, multi-person tracking, and thresholding, and discusses future directions like privacy-preserving multimodal fusion and federated learning. The work aims to catalyze the adoption of privacy-protecting anomaly detection in homes, care facilities, and other sensitive environments, ultimately improving safety and quality of life while maintaining individual privacy.

Abstract

The existing methods for video anomaly detection mostly utilize videos containing identifiable facial and appearance-based features. The use of videos with identifiable faces raises privacy concerns, especially when used in a hospital or community-based setting. Appearance-based features can also be sensitive to pixel-based noise, straining the anomaly detection methods to model the changes in the background and making it difficult to focus on the actions of humans in the foreground. Structural information in the form of skeletons describing the human motion in the videos is privacy-protecting and can overcome some of the problems posed by appearance-based features. In this paper, we present a survey of privacy-protecting deep learning anomaly detection methods using skeletons extracted from videos. We present a novel taxonomy of algorithms based on the various learning approaches. We conclude that skeleton-based approaches for anomaly detection can be a plausible privacy-protecting alternative for video anomaly detection. Lastly, we identify major open research questions and provide guidelines to address them.
Paper Structure (25 sections, 2 figures, 1 table)

This paper contains 25 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Literature Survey
  3. Reconstruction Approaches
  4. Prediction Approaches
  5. Combinations of learning approaches
  6. Combination of reconstruction and prediction approaches
  7. Combination of reconstruction and clustering approaches
  8. Other Approaches
  9. Discussion
  10. Datasets
  11. Number of people in the scene
  12. Fields of application
  13. Choice of pose estimation algorithm
  14. Model type
  15. Evaluation metrics
  16. ...and 10 more sections

Figures (2)

  • Figure 1: One normal and one anomalous frame from each of the skeletal video anomaly detection datasets.
  • Figure 2: Openpose (top row) and alphapose (bottom row) output on different datasets.