Exploring EEG and Eye Movement Fusion for Multi-Class Target RSVP-BCI

TL;DR

This paper tackles the challenge of multi-class RSVP-BCI by integrating EEG with eye movement signals. It introduces MTREE-Net, a two-stream architecture with a dual-complementary module, contribution-guided reweighting, and hierarchical self-distillation to fuse modalities and differentiate multiple target categories. Through an open multi-class RSVP EEG–EM dataset (43 subjects) and extensive ablations, MTREE-Net consistently outperforms EEG-only and existing EEG–EM fusion baselines, with analyses showing strong EEG parietal/occipital and EM pupil-area cues driving discrimination. The work demonstrates the feasibility and advantages of EM integration for practical multi-class RSVP-BCI systems and provides a dataset and code framework to spur future development.

Abstract

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interfaces (BCIs) facilitate high-throughput target image detection by identifying event-related potentials (ERPs) evoked in EEG signals. The RSVP-BCI systems effectively detect single-class targets within a stream of images but have limited applicability in scenarios that require detecting multiple target categories. Multi-class RSVP-BCI systems address this limitation by simultaneously identifying the presence of a target and distinguishing its category. However, existing multi-class RSVP decoding algorithms predominantly rely on single-modality EEG decoding, which restricts their performance improvement due to the high similarity between ERPs evoked by different target categories. In this work, we introduce eye movement (EM) modality into multi-class RSVP decoding and explore EEG and EM fusion to enhance decoding performance. First, we design three independent multi-class target RSVP tasks and build an open-source dataset comprising EEG and EM signals from 43 subjects. Then, we propose the Multi-class Target RSVP EEG and EM fusion Network (MTREE-Net) to enhance multi-class RSVP decoding. Specifically, a dual-complementary module is proposed to strengthen the differentiation of uni-modal features across categories. To improve multi-modal fusion performance, we adopt a dynamic reweighting fusion strategy guided by theoretically derived modality contribution ratios. Furthermore, we reduce the misclassification of non-target samples through knowledge transfer between two hierarchical classifiers. Extensive experiments demonstrate the feasibility of integrating EM signals into multi-class RSVP decoding and highlight the superior performance of MTREE-Net compared to existing RSVP decoding methods. The proposed MTREE-Net and open-source dataset provide a promising framework for developing practical multi-class RSVP-BCI systems.

Figures (9)

• Figure 1: The diagrams of (a) single-class target RSVP-BCI and (b) multi-class target RSVP-BCI. In the single-class target RSVP-BCI, subjects identify target images (e.g., people) within an image sequence. The decoding model detects target presence by identifying ERPs evoked by the target images. In the multi-class target RSVP-BCI, participants identify multiple target categories (e.g., people and cars) within the image sequence. The decoding model detects the presence of targets and differentiates between target categories by identifying and classifying ERPs evoked by different target classes.
• Figure 2: Illustration of the RSVP paradigm. (a) Examples of target-1, target-2, and non-target images in Task A, Task B, and Task C. The stimulus images in three tasks are sourced from the remote sensing Dior dataset li2020object. (b) Experimental settings about the division of tasks, blocks, and sequences for each subject.
• Figure 3: The structure of our proposed MTREE-Net model consists of a two-stream feature extractor, a dual-complementary module (DCM), a contribution-guided reweighting module (CG-RM), and a hierarchical self-distillation module (HSM). The network employs several loss functions: intra-modal triplet cross-entropy losses ($\mathcal{L}_{intra\hbox{-}eeg}$ and $\mathcal{L}_{intra\hbox{-}em}$), binary cross-entropy loss ($\mathcal{L}_{bce}$), triplet cross-entropy loss ($\mathcal{L}_{ce}$), contribution-guided loss with $L_1$-norm ($\mathcal{L}_{cg}$), and self-distillation loss using symmetric Kullback-Leibler divergence ($\mathcal{L}_{sd}$). The output logits $N$, $T$, $T1$, and $T2$ correspond to non-target, target, target-1, and target-2 classes, respectively.
• Figure 4: The confusion matrices of MTREE-Net and the EEG baseline, as well as the differences between the results of MTREE-Net and the EEG baseline for (a) Task A, (b) Task B, and (c) Task C. Each confusion matrix displays all metrics as percentages ($\%$), normalized across rows. The labels T1, T2, and NT denote target-1, target-2, and non-target, respectively. The ${\star}$ denotes that the performance metric of MTREE-Net is significantly higher than that of the EEG baseline. At the same time, the ${\dagger}$ indicates that the performance metric of MTREE-Net is significantly lower than that of the EEG baseline ($p<0.05$).
• Figure 5: The improvement in BA achieved by MTREE-Net and the ablation model (w/o $\mathcal{L}_{cg}$), compared to the optimal uni-modal BA achieved using the uni-modal features ($\boldsymbol{x}_{eeg}$ or $\boldsymbol{x}_{em}$) within the multi-modal model, is shown for (a) Task A, (b) Task B, and (c) Task C. Each scatter represents the averaged performance of a subject in three tasks, with the x-coordinate indicating the ablation model’s improved BA on each subject, and the y-coordinate showing the BA of complete MTREE-Net (w/ $\mathcal{L}_{cg}$). Scatters above the $y=x$ indicate the improved performance of the model with $\mathcal{L}_{cg}$ on the subject is better than the improved performance without $\mathcal{L}_{cg}$.