The role of data partitioning on the performance of EEG-based deep learning models in supervised cross-subject analysis: a preliminary study

TL;DR

This paper addresses how data partitioning and cross-validation influence EEG-based deep learning performance in cross-subject analyses. It systematically compares five cross-validation settings (K-Fold, LNSO, LOSO, N-LNSO, N-LOSO) across three tasks (Parkinson's, Alzheimer's, BCI) and four architectures (ShallowConvNet, EEGNet, DeepConvNet, T-ResNet) using data from OpenNeuro datasets, with over 100,000 trained models. The key finding is that traditional sample-based CV and non-nested subject-based CV inflate performance due to subject-specific leakage, while nested approaches (N-LNSO, N-LOSO) yield more realistic, generalizable estimates, albeit with higher computational cost and lower medians for complex models. The paper offers concrete guidelines on outer/inner fold configurations based on dataset size to balance bias-variance and proposes a forward path toward larger multicenter EEG datasets to improve generalizability and evaluation rigor.

Abstract

Deep learning is significantly advancing the analysis of electroencephalography (EEG) data by effectively discovering highly nonlinear patterns within the signals. Data partitioning and cross-validation are crucial for assessing model performance and ensuring study comparability, as they can produce varied results and data leakage due to specific signal properties (e.g., biometric). Such variability leads to incomparable studies and, increasingly, overestimated performance claims, which are detrimental to the field. Nevertheless, no comprehensive guidelines for proper data partitioning and cross-validation exist in the domain, nor is there a quantitative evaluation of their impact on model accuracy, reliability, and generalizability. To assist researchers in identifying optimal experimental strategies, this paper thoroughly investigates the role of data partitioning and cross-validation in evaluating EEG deep learning models. Five cross-validation settings are compared across three supervised cross-subject classification tasks (BCI, Parkinson's, and Alzheimer's disease detection) and four established architectures of increasing complexity (ShallowConvNet, EEGNet, DeepConvNet, and Temporal-based ResNet). The comparison of over 100,000 trained models underscores, first, the importance of using subject-based cross-validation strategies for evaluating EEG deep learning models, except when within-subject analyses are acceptable (e.g., BCI). Second, it highlights the greater reliability of nested approaches (N-LNSO) compared to non-nested counterparts, which are prone to data leakage and favor larger models overfitting to validation data. In conclusion, this work provides EEG deep learning researchers with an analysis of data partitioning and cross-validation and offers guidelines to avoid data leakage, currently undermining the domain with potentially overestimated performance claims.

Figures (8)

• Figure 1: Schematic representation of the three categories of cross-validation investigated in this work, inspired by DelPup2024b. (a) Sample-based K-Fold (K-Fold) randomly assigns windows of the same EEG both in the training and in the validation set, ignoring the subject; (b) Leave-N-Subjects-Out (LNSO or LOSO if N=1) randomly assigns windows of the same subject either in the training or in the validation set (not in both); (c) Nested-Leave-N-Subjects-Out (N-LNSO or N-LOSO) introduces a nested level for the creation of multiple train-validation splits for each test split. Models are trained on the training set, and evaluated on test set, if available; otherwise, the validation set is used. This procedure generates an ensemble of performances, which is used for the subsequent cross-validation comparisons. The training is controlled using early stopping criteria on the validation set. Note that for (a) and (b), the usage of the validation set to both monitor the training and evaluate the model introduces data leakage, which is, however, considered in the following analysis.
• Figure 1: Balanced accuracy estimation bias comparison between Sample-based (K-Fold) and subject-based (LNSO) 10-Fold cross-validations. The sub-figures display results for different deep learning architectures: ShallowConvNet (Panel A), EEGNet (Panel B), DeepConvNet (Panel C), and T-ResNet (Panel D). Sample-based approaches produce strong positive biases, particularly in pathology classification tasks, indicating the tendency of this method to produce optimistic results and underestimating the generalization error.
• Figure 2: Balanced accuracy comparison between Sample-based (K-Fold, in blue) and subject-based (LNSO, in orange) 10-Fold cross-validations. The sub-figures display results for different deep learning architectures: ShallowConvNet (Panel A), EEGNet (Panel B), DeepConvNet (Panel C), and T-ResNet (Panel D). Results are shown across all tasks. Independently from the model, there is a performance drop and an increased variance when switching to subject-based cross-validation methods, particularly in pathology classification tasks. Additionally, ShallowConvNet, the smallest model, achieved the highest median accuracies, highlighting a potential drawback in the usage of more complex models.
• Figure 3: Comparison of balanced accuracy between Nested-Leave-N-Subjects-Out (N-LNSO) and Leave-N-Subjects-Out (LNSO). The sub-figures display results for different deep learning architectures: ShallowConvNet (Panel A), EEGNet (Panel B), DeepConvNet (Panel C), and T-ResNet (Panel D). Results are shown across all tasks, with light blue representing Parkinson's, green for Alzheimer's, and yellow for BCI. Each column of points shows N-LNSO results for architectures trained on different train/validation partitions (inner folds) but evaluated on the same test set (outer fold), as a function of LNSO accuracies. Notably, performance differences between the two CVs increase with higher LNSO fold accuracies. The regression lines' slopes (in red), always less than 1, further highlight this trend. Additionally, more complex models exhibit greater result variance.
• Figure 4: Subject-wise analysis of Nested-Leave-One-Subject-Out (N-LOSO) results. On the left, the distribution of differences across all subjects between the Leave-One-Subject-Out (LOSO) balanced accuracies and the median N-LOSO is evaluated for each model architecture and classification task. On the right, the interquartile range distribution of N-LOSO results for each subject is evaluated for the same architectures and classification tasks. Both figures highlight minimal differences in the BCI task, but reveal higher performance drops in pathologic tasks for more complex models such as T-ResNet.