Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing

TL;DR

This work tackles cross-dataset generalization in EEG-based emotion recognition by introducing mdJPT, a task-specific multi-dataset pre-training framework. It jointly pre-trains on multiple emotion datasets using cross-dataset covariance alignment (CDA) and inter-subject alignment (ISA) losses, combined with a hybrid MLLA-based channel encoder and a spatiotemporal dynamics module to capture long-range temporal and inter-channel patterns. Empirical results show mdJPT yields state-of-the-art few-shot performance and strong zero-shot generalization across diverse datasets, with performance improving as more datasets are included. The approach offers a scalable paradigm for task-specific pre-training in affective computing and holds potential for broader BCI applications, while also highlighting areas for improvement in fine-grained emotion and cross-context generalization.

Abstract

Task-specific pre-training is essential when task representations diverge from generic pre-training features. Existing task-general pre-training EEG models struggle with complex tasks like emotion recognition due to mismatches between task-specific features and broad pre-training approaches. This work aims to develop a task-specific multi-dataset joint pre-training framework for cross-dataset emotion recognition, tackling problems of large inter-dataset distribution shifts, inconsistent emotion category definitions, and substantial inter-subject variability. We introduce a cross-dataset covariance alignment loss to align second-order statistical properties across datasets, enabling robust generalization without the need for extensive labels or per-subject calibration. To capture the long-term dependency and complex dynamics of EEG, we propose a hybrid encoder combining a Mamba-like linear attention channel encoder and a spatiotemporal dynamics model. Our method outperforms state-of-the-art large-scale EEG models by an average of 4.57% in AUROC for few-shot emotion recognition and 11.92% in accuracy for zero-shot generalization to a new dataset. Performance scales with the increase of datasets used in pre-training. Multi-dataset joint pre-training achieves a performance gain of 8.55% over single-dataset training. This work provides a scalable framework for task-specific pre-training and highlights its benefit in generalizable affective computing. Our code is available at https://github.com/ncclab-sustech/mdJPT_nips2025.

Figures (9)

• Figure 1: Cross-dataset generalization through multi-dataset pre-training. (left) Challenges of task-specific EEG pre-training. (right) Challenges of inference.
• Figure 2: The overall framework, model architectures, and loss designs of mdJPT. A) The overall framework. Our approach comprises three stages: multi-dataset pre-training, optional classifier fine-tuning, and testing. The EEG encoder integrates an MLLA channel encoder with a spatiotemporal dynamics model. Pre-training combines CDA and inter-subject alignment (ISA) losses. The resulting model supports either zero-shot generalization or fine-tuning with a lightweight classifier on limited target data. Final evaluation is performed on held-out subjects. B) The cross-dataset alignment loss. The covariance is calculated from the latent representations, and covariance centroids of subjects from different datasets are aligned. C) The inter-subject alignment loss. EEG samples from two subjects corresponding to the same time segment within a trial are regarded as positive samples, and those corresponding to different trials are regarded as negative samples. A contrastive loss is employed to pull together the positive samples and push apart the negative samples. D) MLLA channel encoder. A channel-independent Mamba-like linear attention encoder is employed to process the time-slided patches of each EEG channel. E) Spatiotemporal dynamics model. The multichannel EEG embeddings are processed by spatial projections and transition extraction to obtain spatiotemporal EEG patterns, then a local attention is employed to estimate dynamic attention weights on these patterns.
• Figure 3: t-sne visualization of features. Compared to DE, pretrained features show better cross-dataset alignment, with features from different datasets more evenly intermixed.
• Figure 4: Generalization performance of mdJPT with an increasing number of datasets. Few-shot setting is used here with SEED-V dataset as the testing dataset. Abbr. S3, S4, S7, F, D stand for SEED, SEED-IV, SEED-VII, FACED, and DEAP, respectively.
• Figure S1: Confusion matrices of emotion classification results on FACED and EmoEEG-imagery datasets.