Uncovering Patterns of Brain Activity from EEG Data Consistently Associated with Cybersickness Using Neural Network Interpretability Maps

TL;DR

The paper tackles real-time cybersickness detection from EEG by leveraging ERP responses to an auditory stimulus to avoid VR-visual confounds. It trains CNN and transformer models (EEGNet, EEG Conformer, and pre-trained EEG ViT), applies calibration to bridge inter-subject variability, and uses interpretability maps (Integrated Gradients and GradCAM) to identify salient features. A consistent finding across models and runs is the prominence of amplitudes at a left prefrontal scalp site (LLPf) for sickness classification, suggesting a transferable biomarker for online detection. The work demonstrates that calibration significantly boosts performance and provides a framework for explainable EEG-based cybersickness classification with potential extensions to time-frequency analysis and graph-based electrode modeling.

Abstract

Cybersickness poses a serious challenge for users of virtual reality (VR) technology. Consequently, there has been significant effort to track its occurrence during VR use with brain activity through electroencephalography (EEG). However, a significant confound in current methods for detecting sickness from EEG is they do not account for the simultaneous processing of the sickening visual stimulus that is present in the brain data from VR. Using event-related potentials (ERPs) from an auditory stimulus shown to reflect cybersickness impacts, we can more precisely target EEG cybersickness features and use those to achieve better performance in online cybersickness classification. In this article, we introduce a method utilizing trained convolutional neural networks and transformer models and plot interpretability maps from integrated gradients and class activation to give a visual representation of what the model determined was most useful in sickness classification from an EEG dataset consisting of ERPs recorded during the elicitation of cybersickness. Across 12 runs of our method with three different neural networks, the models consistently pointed to a surprising finding: that amplitudes recorded at an electrode placed on the scalp near the left prefrontal cortex were important in the classification of cybersickness. These results help clarify a hidden pattern in other related research and point to exciting opportunities for future investigation: that this scalp location could be used as a tagged feature for better real-time cybersickness classification with EEG. We provide our code at: [anonymized].

Figures (10)

• Figure 1: Electrode locations with the front of the head at the top. The electrode names indicate location: R=right, L=left (or L=lateral if second letter), M=medial, D=dorsal, Pf=prefrontal cortex, Ce=centerline, Pa=parietal cortex, Oc=occipital cortex.
• Figure 2: Leaving out Subject 10 from the remaining subjects.
• Figure 3: K-means centroids of 6 clusters and average over all data.
• Figure 4: t-SNE for 2 components. Left: data points from the same cluster have the same color. Right: data points are colored based on sick and non-sick label.
• Figure 5: Two-phase process: 1) main training (A) for leave-one-subject-out 1-fold and 2) calibration (B and C) to fine-tune model to the test subject. There are two possible calibration datasets depending on if the test subject data has both labels or not. For main training, the train set goes through outlier detection, z-score normalization, and random oversampling. Best validation checkpoint is fine-tuned in calibration. If test subject has both labels, then 25% of the test subject's data is sampled. If test subject only has one label, then 12.5% of test subject's data is sampled and top k data of the other label from other subjects are taken that are closest to sampled test subject data centroid in PCA space.