Learning Exemplar Representations in Single-Trial EEG Category Decoding
Jack Kilgallen, Barak Pearlmutter, Jeffery Mark Siskind
TL;DR
This work addresses the risk of exemplar leakage in single-trial EEG category decoding when trials from the same stimulus appear in both training and testing. By creating pseudocategories with meaningless labels in two public EEG datasets (Kaneshiro 2015 and Gifford 2022) and evaluating a range of classifiers, including simple k-NN and deep networks, the authors demonstrate that models can learn exemplar-level representations from category labels alone. All models achieved decoding above chance, even under balanced pseudocategories and across datasets with different exemplar counts, suggesting that prior high accuracies in the literature may be inflated by leakage. The study highlights the need to reevaluate reported single-trial EEG decoding results and to adopt evaluation schemes that prevent exemplar leakage, with implications for neuroimaging research and BCI applications. This work also shows that exemplar leakage can persist even when the number of exemplars per category is large, underscoring the uncertain state-of-the-art in this domain.
Abstract
Within neuroimgaing studies it is a common practice to perform repetitions of trials in an experiment when working with a noisy class of data acquisition system, such as electroencephalography (EEG) or magnetoencephalography (MEG). While this approach can be useful in some experimental designs, it presents significant limitations for certain types of analyses, such as identifying the category of an object observed by a subject. In this study we demonstrate that when trials relating to a single object are allowed to appear in both the training and testing sets, almost any classification algorithm is capable of learning the representation of an object given only category labels. This ability to learn object representations is of particular significance as it suggests that the results of several published studies which predict the category of observed objects from EEG signals may be affected by a subtle form of leakage which has inflated their reported accuracies. We demonstrate the ability of both simple classification algorithms, and sophisticated deep learning models, to learn object representations given only category labels. We do this using two datasets; the Kaneshiro et al. (2015) dataset and the Gifford et al. (2022) dataset. Our results raise doubts about the true generalizability of several published models and suggests that the reported performance of these models may be significantly inflated.