Semi-Supervised Dual-Stream Self-Attentive Adversarial Graph Contrastive Learning for Cross-Subject EEG-based Emotion Recognition

TL;DR

The paper tackles label scarcity in cross-subject EEG emotion recognition by presenting DS-AGC, a semi-supervised framework that fuses non-structural and structural EEG features through a dual-stream architecture and self-attentive fusion. It jointly aligns three domains, $\mathbb{S}$, $\mathbb{U}$, and $\mathbb{T}$, via a multi-domain adversarial loss and leverages graph contrastive learning on EEG channels to capture inter-channel structure. Experimental results across SEED, SEED-IV, SEED-V, and FACED demonstrate robust improvements under incomplete labeling, with strong generalization to unseen targets and clear ablation-supported contributions from each module. The approach provides a scalable, practically applicable solution for semi-supervised cross-subject EEG emotion recognition with potential impact on affective BCIs and beyond.

Abstract

Electroencephalography (EEG) is an objective tool for emotion recognition with promising applications. However, the scarcity of labeled data remains a major challenge in this field, limiting the widespread use of EEG-based emotion recognition. In this paper, a semi-supervised Dual-stream Self-Attentive Adversarial Graph Contrastive learning framework (termed as DS-AGC) is proposed to tackle the challenge of limited labeled data in cross-subject EEG-based emotion recognition. The DS-AGC framework includes two parallel streams for extracting non-structural and structural EEG features. The non-structural stream incorporates a semi-supervised multi-domain adaptation method to alleviate distribution discrepancy among labeled source domain, unlabeled source domain, and unknown target domain. The structural stream develops a graph contrastive learning method to extract effective graph-based feature representation from multiple EEG channels in a semi-supervised manner. Further, a self-attentive fusion module is developed for feature fusion, sample selection, and emotion recognition, which highlights EEG features more relevant to emotions and data samples in the labeled source domain that are closer to the target domain. Extensive experiments conducted on two benchmark databases (SEED and SEED-IV) using a semi-supervised cross-subject leave-one-subject-out cross-validation evaluation scheme show that the proposed model outperforms existing methods under different incomplete label conditions (with an average improvement of 5.83% on SEED and 6.99% on SEED-IV), demonstrating its effectiveness in addressing the label scarcity problem in cross-subject EEG-based emotion recognition.

Figures (6)

• Figure 1: An overview of the proposed DS-AGC. It consists of three parts: non-structural stream, structural stream, and self-attentive fusion. $\mathbb{S}$, $\mathbb{U}$, and $\mathbb{T}$ refer to the labeled source domain, unlabeled source domain, and unknown target domain.
• Figure 2: The cross-subject leave-one-subject-out cross-validation experimental protocol with incomplete labels. $\mathbb{S}$, $\mathbb{U}$, and $\mathbb{T}$ represent the labeled source domain, unlabeled source domain, and unknown target domain. For the SEED and SEED-IV databases, which contain a total of 15 subjects, $14 - N$ subjects are allocated to $\mathbb{S}$, and $N$ subjects to $\mathbb{U}$. Similarly, for the SEED-V database with 16 subjects, $15 - N$ subjects are assigned to $\mathbb{S}$, and $N$ subjects to $\mathbb{U}$. $M$ denotes the total number of trials for each subject. $\mathcal{L}_{ce}$, $\mathcal{L}_{gcn}$, $\mathcal{L}_{gcl}$, and $\mathcal{L}_{disc}$ are the classification loss, GCN loss, GCL loss, and discriminator loss, given in Eq. \ref{['Eq:class']}, Eq. \ref{['Eq:aij_loss']}, Eq. \ref{['Eq:gcl']}, and Eq. \ref{['Eq:Tripledomainloss']}. In the implementation, $\mathcal{L}_{ce}$ is calculated using only $\mathbb{S}$ (represented by blue stars), as the label information for $\mathbb{U}$ and $\mathbb{T}$ is unknown. $\mathcal{L}_{gcn}$, $\mathcal{L}_{gcl}$, and $\mathcal{L}_{disc}$, which do not depend on label information, are calculated using data from $\mathbb{S}$ (blue stars), $\mathbb{U}$ (pink circles), and $\mathbb{T}$ (gray triangles).
• Figure 3: Experimental results under different settings on the SEED databases. Red line with triangle marker: the proposed DS-AGC; pink line with circle marker: single structural stream; green line with square marker: single non- structural stream; blue line with diamond marker: without unlabeled source data.
• Figure 4: Model performance under various $E_t$ values on the SEED database.
• Figure 5: A visualization of the obtained informative feature $\{\xi_{1}, \ldots, \xi_{\dot{m}+\dot{n}}\}$ by $MHA(\cdot)$ at three different stages: (a) before training, (b) at the 30th training epoch, and (c) in the final trained model. In this visualization, the circle, asterisk, and triangle represent the labeled source domain ($\mathbb{S}$), unlabeled source domain ($\mathbb{U}$), and the unknown target domain ($\mathbb{T}$), respectively. The red, purple, and blue colors correspond to negative, neutral, and positive emotions, respectively.