SeFEF: A Seizure Forecasting Evaluation Framework

TL;DR

SeFEF addresses the lack of standardized evaluation in seizure forecasting by offering a Python-based framework that automates data labeling, time-series cross-validation, forecast post-processing, and reporting, complemented by model cards for documentation. It supports probabilistic forecasts $P(S=1|X)$ across forecast horizons $T_H$ and reports both deterministic metrics (e.g., Sen, FPR, TiW) and probabilistic scores (e.g., BS, BSS, reliability, resolution). Three proof-of-concept models illustrate integration of time-series features and seizure periodicity: a Von Mises estimator, a standard LR, and a periodicity-aware LR ensemble that updates intercepts with periodic priors. While the validation is limited to a single-user, SeFEF aims to standardize methodology, reduce development time, and enable community-driven validation across datasets.

Abstract

The lack of standardization in seizure forecasting slows progress in the field and limits the clinical translation of forecasting models. In this work, we introduce a Python-based framework aimed at streamlining the development, assessment, and documentation of individualized seizure forecasting algorithms. The framework automates data labeling, cross-validation splitting, forecast post-processing, performance evaluation, and reporting. It supports various forecasting horizons and includes a model card that documents implementation details, training and evaluation settings, and performance metrics. Three different models were implemented as a proof-of-concept. The models leveraged features extracted from time series data and seizure periodicity. Model performance was assessed using time series cross-validation and key deterministic and probabilistic metrics. Implementation of the three models was successful, demonstrating the flexibility of the framework. The results also emphasize the importance of careful model interpretation due to variations in probability scaling, calibration, and subject-specific differences. Although formal usability metrics were not recorded, empirical observations suggest reduced development time and methodological consistency, minimizing unintentional variations that could affect the comparability of different approaches. As a proof-of-concept, this validation is inherently limited, relying on a single-user experiment without statistical analyses or replication across independent datasets. At this stage, our objective is to make the framework publicly available to foster community engagement, facilitate experimentation, and gather feedback. In the long term, we aim to contribute to the establishment of a consensus on a standardized methodology for the development and validation of seizure forecasting algorithms in people with epilepsy.

Figures (11)

• Figure 1: Illustration of the steps implied in the task of seizure forecasting. The rose boxes (on the right) correspond to steps that are automated by SeFEF, while the blue boxes (on the left) are implemented by the user. Acronyms: CVFolds: # of folds in CV.
• Figure 2: Illustration of TSCV applied to the context of seizure forecasting.
• Figure 3: Effect of post-processing methodology using the optimal prediction case (i.e., using the actual labels as predicted probabilities). A vertical black bar identifies the onset of the seizure.
• Figure 4: Illustration of 2 types of input data that can be represented from a multimodal and long-term dataset (as illustrated in (a)): (b) continuous, multimodal time series; and (c) sporadic, timestamp-only, seizure onset records.
• Figure 5: TSCV for (a) time series dataset and (b) seizure timestamps dataset, for patient 1876, achieved with the evaluation module from . Default parameters were used. Star symbols denote the onset of a seizure, where the ones with lower opacity indicate onsets for which no pre-ictal data was found.