NeuroIDBench: An Open-Source Benchmark Framework for the Standardization of Methodology in Brainwave-based Authentication Research

TL;DR

The paper tackles reproducibility and comparability gaps in brainwave-based authentication by introducing NeuroIDBench, an open-source benchmark framework that integrates nine ERP datasets, flexible preprocessing, feature extraction, and evaluation pipelines under two attacker models. It systematically compares shallow classifiers (e.g., SVM, RF, KNN, LDA, NB, LR) and a similarity-based Twin Neural Network approach, across single- and multi-session settings, highlighting PSD with AR order 1 as a robust default for shallow methods and the mixed, data-dependent performance of deep learning. Key findings show that unknown attackers substantially elevate $EER$ relative to known attackers, multi-session evaluations are significantly more challenging, and deep learning does not consistently outperform traditional feature-based approaches due to limited data. The work demonstrates the value of standardized, open benchmarking for fair comparisons and methodological refinement, and it calls for larger, multi-session, privacy-aware EEG datasets to advance practical brainwave authentication systems.

Abstract

Biometric systems based on brain activity have been proposed as an alternative to passwords or to complement current authentication techniques. By leveraging the unique brainwave patterns of individuals, these systems offer the possibility of creating authentication solutions that are resistant to theft, hands-free, accessible, and potentially even revocable. However, despite the growing stream of research in this area, faster advance is hindered by reproducibility problems. Issues such as the lack of standard reporting schemes for performance results and system configuration, or the absence of common evaluation benchmarks, make comparability and proper assessment of different biometric solutions challenging. Further, barriers are erected to future work when, as so often, source code is not published open access. To bridge this gap, we introduce NeuroIDBench, a flexible open source tool to benchmark brainwave-based authentication models. It incorporates nine diverse datasets, implements a comprehensive set of pre-processing parameters and machine learning algorithms, enables testing under two common adversary models (known vs unknown attacker), and allows researchers to generate full performance reports and visualizations. We use NeuroIDBench to investigate the shallow classifiers and deep learning-based approaches proposed in the literature, and to test robustness across multiple sessions. We observe a 37.6% reduction in Equal Error Rate (EER) for unknown attacker scenarios (typically not tested in the literature), and we highlight the importance of session variability to brainwave authentication. All in all, our results demonstrate the viability and relevance of NeuroIDBench in streamlining fair comparisons of algorithms, thereby furthering the advancement of brainwave-based authentication through robust methodological practices.

Figures (19)

• Figure 1: Overview of elements in a brainwave-based biometric authentication process arias2023performance.
• Figure 2: Schematic depiction outlining the architectural structure of the NeuroIDBench codebase, providing a visual overview of its underlying framework jayaram2018moabb
• Figure 3: The CNN's network architecture within the Twin neural Network is designed to produce a condensed 32-bit embedding from brain data samples, serving as an efficient method to compute latent representations of inputs fallahi2023brainnet.
• Figure 4: Impact of varying sample duration, ranging from 1 to 2 seconds, on the efficacy of the seven authentication algorithms. These algorithms are applied across nine datasets, with the evaluation metric employed being the EER. The evaluation was conducted under the unknown attacker scenario.
• Figure 5: Influence of varying sample rejection thresholds, spanning from 100 to 400 microvolts, as well as scenarios with no rejection, on the performance outcomes of seven authentication algorithms. The investigation is conducted across nine distinct datasets, employing the EER as the metric for comprehensive performance assessment, and performance assessment is done under an unknown attacker scenario.