Uncertainty Quantification for cross-subject Motor Imagery classification

TL;DR

This work applied a variety of Uncertainty Quantification methods to predict misclassifications for a Motor Imagery Brain Computer Interface, and found that standard CNNs with Softmax output performed better than some of the more advanced methods.

Abstract

Uncertainty Quantification aims to determine when the prediction from a Machine Learning model is likely to be wrong. Computer Vision research has explored methods for determining epistemic uncertainty (also known as model uncertainty), which should correspond with generalisation error. These methods theoretically allow to predict misclassifications due to inter-subject variability. We applied a variety of Uncertainty Quantification methods to predict misclassifications for a Motor Imagery Brain Computer Interface. Deep Ensembles performed best, both in terms of classification performance and cross-subject Uncertainty Quantification performance. However, we found that standard CNNs with Softmax output performed better than some of the more advanced methods.

Figures (2)

• Figure 1: An illustration of a discriminative and a generative model. The yellow and purple dots indicate the training samples of two different classes. The background indicates the prediction. The green color indicates uncertainty.
• Figure 2: Training setup for a single subject. One subject is excluded and used as an out-of-population set while the other 10% of the data from each subject is separated into a within-population set. The data of the remaining subjects are concatenated and split 90-10 into a training and validation set. This procedure is repeated for every subject.