Deep Learning Approaches for Seizure Video Analysis: A Review

TL;DR

The foundation technologies used in vision-based systems in the analysis of seizure videos are detailed, highlighting their success in semiology detection and analysis, focusing on work published in the last 7 years.

Abstract

Seizure events can manifest as transient disruptions in the control of movements which may be organized in distinct behavioral sequences, accompanied or not by other observable features such as altered facial expressions. The analysis of these clinical signs, referred to as semiology, is subject to observer variations when specialists evaluate video-recorded events in the clinical setting. To enhance the accuracy and consistency of evaluations, computer-aided video analysis of seizures has emerged as a natural avenue. In the field of medical applications, deep learning and computer vision approaches have driven substantial advancements. Historically, these approaches have been used for disease detection, classification, and prediction using diagnostic data; however, there has been limited exploration of their application in evaluating video-based motion detection in the clinical epileptology setting. While vision-based technologies do not aim to replace clinical expertise, they can significantly contribute to medical decision-making and patient care by providing quantitative evidence and decision support. Behavior monitoring tools offer several advantages such as providing objective information, detecting challenging-to-observe events, reducing documentation efforts, and extending assessment capabilities to areas with limited expertise. The main applications of these could be (1) improved seizure detection methods; (2) refined semiology analysis for predicting seizure type and cerebral localization. In this paper, we detail the foundation technologies used in vision-based systems in the analysis of seizure videos, highlighting their success in semiology detection and analysis, focusing on work published in the last 7 years. Additionally, we illustrate how existing technologies can be interconnected through an integrated system for video-based semiology analysis.

Figures (4)

• Figure 1: Overview of deep learning applications for semiology analysis. A. Motion capture and skeleton-based human action recognition. B. Frame-based human action recognition. C. Feature representations of the motion exhibited by a patient during a seizure, which is used to identify if new data samples are similar to known semiologies, enabling the categorization of Epilepsy types.
• Figure 2: Overview of skeleton-based human action recognition. A. Motion capture: Includes 2D/3D landmarks and 3D mesh estimation for the body, face and hands. B. Spatio-temporal representation: depicts the motion of each keypoint from the body, face, or hands. RNN-based, CNN-based, GNN-based, and Transformer-based methods are all commonly used with this type of approach.
• Figure 3: Overview of frame-based human action recognition. A. Motion capture: Region of interest detection such as the body, face, or hands. B. Spatio-temporal representations are learned from sequential frames. CNN-RNN-based, 3D-CNN-based and Transformer-based methods are all commonly used with this type of approach.
• Figure 4: Overview of an automatic seizure video evaluation system. A potential implementation could be a closed-loop system designed for clinical deployment in the hospital that integrates: A. A flexible and modular markerless motion capture solution, B. A method enabling the inspection of the stepwise progression of clinical features and the order of signs, C. An analysis of the whole body to distinguish seizure disorders such as PNES, D. A flexible and modular multi-stream framework to evaluate isolated semiologies, E. An inspection of the classification results and model uncertainty (model confidence), as well as a set of model-agnostic methods to interpret deep learning methodologies, and F. An anomaly detection module to update the system with aberrant video features of semiology. ES: Epileptic seizures; FND: functional neurological disorder; PNES: psychogenic non-epileptic seizures.