Error-related Potential driven Reinforcement Learning for adaptive Brain-Computer Interfaces

TL;DR

This work addresses EEG non-stationarities in non-invasive BCIs by coupling motor imagery with error-related potentials (ErrPs) through a reinforcement-learning framework. It validates an ErrP-driven contextual-bandit approach using two agents, LinUCB and NeuralUCB, on an open MI dataset and a novel in-house fast-paced MI+ErrP snake game, extracting time-frequency features via continuous wavelet transform and using ErrP presence as the reward signal. Results show that the RL agents can learn mappings from MI features to actions in offline settings, with performance highly dependent on MI separability and feature quality; some subjects reach near-chance performance, while others achieve robust results, and a pilot in-house task reveals that high-speed MI tasks may reduce discriminability. The findings underscore the potential of RL-based adaptive BCIs driven by ErrPs, while also identifying practical constraints such as data quality, task design, and the need for online validation and robust ErrP classification for real-time deployment.

Abstract

Brain-computer interfaces (BCIs) provide alternative communication methods for individuals with motor disabilities by allowing control and interaction with external devices. Non-invasive BCIs, especially those using electroencephalography (EEG), are practical and safe for various applications. However, their performance is often hindered by EEG non-stationarities, caused by changing mental states or device characteristics like electrode impedance. This challenge has spurred research into adaptive BCIs that can handle such variations. In recent years, interest has grown in using error-related potentials (ErrPs) to enhance BCI performance. ErrPs, neural responses to errors, can be detected non-invasively and have been integrated into different BCI paradigms to improve performance through error correction or adaptation. This research introduces a novel adaptive ErrP-based BCI approach using reinforcement learning (RL). We demonstrate the feasibility of an RL-driven adaptive framework incorporating ErrPs and motor imagery. Utilizing two RL agents, the framework adapts dynamically to EEG non-stationarities. Validation was conducted using a publicly available motor imagery dataset and a fast-paced game designed to boost user engagement. Results show the framework's promise, with RL agents learning control policies from user interactions and achieving robust performance across datasets. However, a critical insight from the game-based protocol revealed that motor imagery in a high-speed interaction paradigm was largely ineffective for participants, highlighting task design limitations in real-time BCI applications. These findings underscore the potential of RL for adaptive BCIs while pointing out practical constraints related to task complexity and user responsiveness.

Figures (10)

• Figure 1: Overview of the BCI framework using ErrP and reinforcement learning. We consider non-invasive BCIs using EEG for neural signal acquisition. As proof-of-concept, we include BCIs based on motor imagery paradigms. The ErrP information is used as reward for the RL agent (3), which learns the mapping between motor imagery input features (1) and corresponding action (2) while subject plays the modified snake game.
• Figure 2: The experimental task: the subjects played the game by imagining a hand movement to control the snake and avoid collision with itself while following the given path (the displayed dots) to collect as many points as possible. In each trial in which the subject's input command was expected, with a probability of $5$%, the snake moved in the wrong direction (as depicted in Trial $2$) to keep subjects motivated, but also to elicit ErrPs.
• Figure 3: Performance of two contextual bandit agents (LinUCB and NeuralUCB) in the evaluation sessions for the open source dataset (n = 9). Results are averaged over different seeds (five and ten, respectively). The plots show the accumulated number of errors across all trials. The accuracy is calculated based on the final accumulated regret. Results show that both agents perform reasonably well for all except two subjects (B02, B03). There is no statistically significant difference in the performance of both agents (two-sided wilcoxon signed rank test, p = 0.91).
• Figure 4: Event-related spectral perturbation (ERSP) for one subject ($S07$) at channels C3 and C4 for both left and right hand motor imagery tasks. The color bars show the color and power spectral density in dB. For each channel, we used EEGLAB to compare the two experimental conditions (left vs right) and show in the right-most plot the permutation results (800 permutations, using false-discovery rate correction for multiple comparisons, significant p-values shown in red for $\alpha=0.05$). These plots highlight how motor-imagery related spectral modulations could be measured for this subject, with modulations mostly visible in the frequency ranges of (10-13) Hz and (16-30) Hz.
• Figure 5: Event-related spectral perturbation (ERSP) for all subjects (n = 7) at channels C3 and C4 for both left and right hand motor imagery tasks. The color bars show the color and power spectral density in dB. For each channel, we used EEGLAB to compare the two experimental conditions (left vs right) and show in the right-most plot the permutation results (800 permutations, using false-discovery rate correction for multiple comparisons, for $\alpha=0.05$). These plots show that, some ERD is visible, the differences across experimental conditions are not significant when considering all subjects.