Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SzCORE: A Seizure Community Open-source Research Evaluation framework for the validation of EEG-based automated seizure detection algorithms

Jonathan Dan, Una Pale, Alireza Amirshahi, William Cappelletti, Thorir Mar Ingolfsson, Xiaying Wang, Andrea Cossettini, Adriano Bernini, Luca Benini, Sándor Beniczky, David Atienza, Philippe Ryvlin

TL;DR

A unified framework designed to establish standardization in the validation of EEG‐based seizure detection algorithms is proposed and the EEG 10–20 seizure detection benchmark is proposed, a machine‐learning benchmark based on public datasets converted to a standardized format.

Abstract

The need for high-quality automated seizure detection algorithms based on electroencephalography (EEG) becomes ever more pressing with the increasing use of ambulatory and long-term EEG monitoring. Heterogeneity in validation methods of these algorithms influences the reported results and makes comprehensive evaluation and comparison challenging. This heterogeneity concerns in particular the choice of datasets, evaluation methodologies, and performance metrics. In this paper, we propose a unified framework designed to establish standardization in the validation of EEG-based seizure detection algorithms. Based on existing guidelines and recommendations, the framework introduces a set of recommendations and standards related to datasets, file formats, EEG data input content, seizure annotation input and output, cross-validation strategies, and performance metrics. We also propose the 10-20 seizure detection benchmark, a machine-learning benchmark based on public datasets converted to a standardized format. This benchmark defines the machine-learning task as well as reporting metrics. We illustrate the use of the benchmark by evaluating a set of existing seizure detection algorithms. The SzCORE (Seizure Community Open-source Research Evaluation) framework and benchmark are made publicly available along with an open-source software library to facilitate research use, while enabling rigorous evaluation of the clinical significance of the algorithms, fostering a collective effort to more optimally detect seizures to improve the lives of people with epilepsy.

SzCORE: A Seizure Community Open-source Research Evaluation framework for the validation of EEG-based automated seizure detection algorithms

TL;DR

A unified framework designed to establish standardization in the validation of EEG‐based seizure detection algorithms is proposed and the EEG 10–20 seizure detection benchmark is proposed, a machine‐learning benchmark based on public datasets converted to a standardized format.

Abstract

The need for high-quality automated seizure detection algorithms based on electroencephalography (EEG) becomes ever more pressing with the increasing use of ambulatory and long-term EEG monitoring. Heterogeneity in validation methods of these algorithms influences the reported results and makes comprehensive evaluation and comparison challenging. This heterogeneity concerns in particular the choice of datasets, evaluation methodologies, and performance metrics. In this paper, we propose a unified framework designed to establish standardization in the validation of EEG-based seizure detection algorithms. Based on existing guidelines and recommendations, the framework introduces a set of recommendations and standards related to datasets, file formats, EEG data input content, seizure annotation input and output, cross-validation strategies, and performance metrics. We also propose the 10-20 seizure detection benchmark, a machine-learning benchmark based on public datasets converted to a standardized format. This benchmark defines the machine-learning task as well as reporting metrics. We illustrate the use of the benchmark by evaluating a set of existing seizure detection algorithms. The SzCORE (Seizure Community Open-source Research Evaluation) framework and benchmark are made publicly available along with an open-source software library to facilitate research use, while enabling rigorous evaluation of the clinical significance of the algorithms, fostering a collective effort to more optimally detect seizures to improve the lives of people with epilepsy.
Paper Structure (36 sections, 5 figures, 6 tables)

This paper contains 36 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. SzCORE framework
  3. EEG Datasets and data format
  4. Datasets
  5. EEG data format:
  6. Seizure annotation format:
  7. Evaluation methodology
  8. Personalized models
  9. Subject-independent models
  10. Performance metrics
  11. Sample-based scoring
  12. Event-based scoring
  13. Performance metrics:
  14. Benchmark
  15. Benchmark datasets
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Time series cross-validation for personalized models. Each box represents an epoch of data. fooorange!50 Orange boxes are used for training, foopurple!50 purple boxes are used for testing. Each row represents a cross-validation fold. The final results are calculated by appending all cross-validation folds (shown in the last row). a) cross-validation scheme with variable amount of data. b) cross-validation scheme with fixed amount of data.
  • Figure 2: Sample-based scoring compares annotation labels sample by sample. Correct detections foogreen!30 (True Positives), false detections foored!30 (False Positives), missed detections fooblue!30 (False Negatives). Seizure annotations are indicated in purple.
  • Figure 3: Event-based scoring is based on overlap. It defines a set of rules for event merging, tolerance before and after events, and maximum event duration. Correct detections foogreen!30 (True Positives), false detections foored!30 (False Positives). Seizure annotations are indicated in purple
  • Figure 4: ILAE 2017 Classification of seizure types (expanded version) Scheffer2017ILAEfoopurple!50 Items in purple are used as short codes to describe an event. As an example a generalized tonic-clonic seizure would be given the code: sz-gen-m-tonic_clonic.
  • Figure : Graphical abstract