Multi-Source EEG Emotion Recognition via Dynamic Contrastive Domain Adaptation

TL;DR

The paper tackles EEG-based emotion recognition under challenging non-stationarity across individuals and sessions. It presents MS-DCDA, a multi-source dynamic contrastive domain adaptation framework that performs coarse inter-domain alignment and fine-grained intra-class alignment via a multi-branch contrastive network, coupled with a class-aware subdomain discrepancy and dynamic loss weighting. Key contributions include the introduction of a source ensemble with per-source branches, a class-aware alignment objective (SCD), and a dynamically weighted loss that balances domain transferability and discriminability, yielding state-of-the-art cross-subject and cross-session results on SEED and SEED-IV (e.g., $90.84\%$ and $78.49\%$ mean accuracy in cross-subject; $95.82\%$ and $82.25\%$ in cross-session). The work also provides insights into frontal and parietal lobe involvement in emotion processing and demonstrates robust generalization, with potential impact on HCI/BCI and personalized mental health interventions.

Abstract

Electroencephalography (EEG) provides reliable indications of human cognition and mental states. Accurate emotion recognition from EEG remains challenging due to signal variations among individuals and across measurement sessions. We introduce a multi-source dynamic contrastive domain adaptation method (MS-DCDA) based on differential entropy (DE) features, in which coarse-grained inter-domain and fine-grained intra-class adaptations are modeled through a multi-branch contrastive neural network and contrastive sub-domain discrepancy learning. Leveraging domain knowledge from each individual source and a complementary source ensemble, our model uses dynamically weighted learning to achieve an optimal tradeoff between domain transferability and discriminability. The proposed MS-DCDA model was evaluated using the SEED and SEED-IV datasets, achieving respectively the highest mean accuracies of $90.84\%$ and $78.49\%$ in cross-subject experiments as well as $95.82\%$ and $82.25\%$ in cross-session experiments. Our model outperforms several alternative domain adaptation methods in recognition accuracy, inter-class margin, and intra-class compactness. Our study also suggests greater emotional sensitivity in the frontal and parietal brain lobes, providing insights for mental health interventions, personalized medicine, and preventive strategies.

Figures (5)

• Figure 1: Comparison of traditional discrepancy-based domain adaptation methods with our model. (a) Single-source domain adaptation (SS-DA) treats data from different subjects as a single source (S) and aligns target (T) with S, ignoring non-stationarity among individual sources. (b) Multi-source domain adaptation (MS-DA) aligns T with each individual source, but tends to produce sub-optimal results due to the lack of fine-grained alignment and imbalanced domain transferability and discriminability. (c) Our multi-source dynamic contrastive domain adaptation (MS-DCDA) model adapts T to individual sources and a complementary multi-source ensemble (SE) with class-awareness. Dynamically weighted domain transferability and discriminability provides improved accuracies, wider inter-class margin, and higher intra-class compactness.
• Figure 2: The pipeline of EEG-based emotion recognition and the schematics of the proposed MS-DCDA model. The multi-source EEG data are measured and preprocessed. The five-band differential entropy (DE) features are extracted from the data of each participant. The multi-source dynamic contrastive domain adaptation (MS-DCDA) model consists of three modules: the common feature extractor (CFE), the multi-branch contrastive (MBC) module, and the multi-branch domain classifier (MBDC). The CFE module extracts features extract domain-invariant features from source data $\{x_s^i| i=0,...N\}$ and the target data $x_t$ using a shared MLP. The MBC and MBDC modules are respectively comprised of $N+1$ independent branches $\{MBC_i\}$ and $\{MBDC_i\}$. The MBC module extracts the domain-variant features and the DC module classify them into domain-specific labels. During training, the $\mathcal{L}_{MMD}$ and $\mathcal{L}_{SCD}$ losses encourage class-independent and class-aware alignment respectively. The classification loss is comprised of the cross entropy loss $\mathcal{L}_{CE}$ and a complementary $\mathcal{L}_{DISC}$ loss, which encourages the predictions consistency across classifiers. Additionally, a dynamic coefficient $\tau$ optimizes the domain transferability and discriminability. During the test, the predicted target class probabilities are averaged across the classifiers, and the target emotion $\hat{e}_t$ is determined by the maximum mean class probability. We train our model in an unsupervised setting, where the source domain is labeled whereas the labels are unavailable for samples in the target domain.
• Figure 3: Confusion matrices analysis of our MS-DCDA model for (a) Cross-subject experiment on SEED dataset; (b) Cross-subject experiment on SEED-IV dataset; (c) Cross-session experiment on SEED; and (d) Cross-session experiment on SEED-IV. Sensitivity are the strongest to the positive and neutral emotions in SEED and SEED-IV dataset, with slightly reduced sensitivity to the rest emotions.
• Figure 4: T-SNE illustration of domain adaptation and emotion recognition on SEED dataset. (a) Distribution of 14 individual source subjects (S1-S14), the ensemble of the 14 individual sources (S15, source ensemble) and a target subject. (b) Distribution of the single source (S) and the target learned by DAN li2018cross. (c) Distribution of the 14 individual source subjects and the target subject learned by our MS-DCDA. (d) The Distribution of the source ensemble and the target. (e) Distribution of the single source and the target learned by DAN. (f) Distribution of the source ensemble and the target learned by our MS-DCDA.
• Figure 5: Comparison of emotion recognition accuracies of different domain adaptation algorithms on SEED and SEED-IV datasets respectively for (a) Cross-subject experiment; and (b) Cross-session experiment. Our MS-DCDA model outperforms in all experiments four single-source methods: DCORAL sun2016deep, DAN li2018cross, DANN kang2020contrastive, DDC tzeng2014deep. The performance of our model also exceeds two multi-source methods: MS-MDA chen2021ms, and MS-ADA she2023multisource.