The Challenge of Out-Of-Distribution Detection in Motor Imagery BCIs

Abstract

Machine Learning classifiers used in Brain-Computer Interfaces make classifications based on the distribution of data they were trained on. When they need to make inferences on samples that fall outside of this distribution, they can only make blind guesses. Instead of allowing random guesses, these Out-of-Distribution (OOD) samples should be detected and rejected. We study OOD detection in Motor Imagery BCIs by training a model on some classes and observing whether unfamiliar classes can be detected based on increased uncertainty. We test seven different OOD detection techniques and one more method that has been claimed to boost the quality of OOD detection. Our findings show that OOD detection for Brain-Computer Interfaces is more challenging than in other machine learning domains due to the high uncertainty inherent in classifying EEG signals. For many subjects, uncertainty for in-distribution classes can still be higher than for out-of-distribution classes. As a result, many OOD detection methods prove to be ineffective, though MC Dropout performed best. Additionally, we show that high in-distribution classification performance predicts high OOD detection performance, suggesting that improved accuracy can also lead to improved robustness. Our research demonstrates a setup for studying how models deal with unfamiliar EEG data and evaluates methods that are robust to these unfamiliar inputs. OOD detection can improve the overall safety and reliability of BCIs.

Figures (8)

• Figure 1: Concept of out-of-distribution detection. A model is trained on several known classes. It has to be able to recognize when unknown data does not belong to any of the known classes, while still classifying the known classes correctly.
• Figure 2: A visual explanation of three different OOD detection approaches. The data points belong to user nine of the BNCI2014 dataset. In this illustration HTML]D55E00feet is considered OOD on a model trained on the HTML]E6CD00right hand and HTML]440154left hand classes.
• Figure 3: Performance of OOD detection methods across three datasets. Colours indicate the following UQ categories: Bayesian , Density , Distance , Baseline .
• Figure 4: Performance of OOD detection for different classes plotted against the different UQ methods."Left", "Right" and"Both" refer to hand movements. Whilst the performance for the Left and Right hand classes stays relatively similar, we find a significant increase in AUROC for the Both hands and Feet classes.
• Figure 5: OOD detection vs on-task classification performance on Bayesian approaches showing a non-linear correlation on the Stieger2021 dataset.