A Brain Wave Encodes a Thousand Tokens: Modeling Inter-Cortical Neural Interactions for Effective EEG-based Emotion Recognition

TL;DR

This work tackles EEG-based emotion recognition by addressing the limitation of electrode-wise approaches that ignore inter-regional brain dynamics. It introduces RBTransformer, a Transformer that tokenizes EEG signals into Band Differential Entropy tokens, augments them with Electrode Identity Embeddings, and applies inter-cortical multi-head attention to model cross-regional neural interactions. The model achieves state-of-the-art results on SEED, DEAP, and DREAMER across Valence, Arousal, and Dominance in both binary and multi-class settings, with comprehensive analyses including t-SNE visualizations and confusion matrices. The findings underscore the importance of inter-cortical dynamics for robust, high-precision EEG-based affective computing, with implications for real-time BCI applications.

Abstract

Human emotions are difficult to convey through words and are often abstracted in the process; however, electroencephalogram (EEG) signals can offer a more direct lens into emotional brain activity. Recent studies show that deep learning models can process these signals to perform emotion recognition with high accuracy. However, many existing approaches overlook the dynamic interplay between distinct brain regions, which can be crucial to understanding how emotions unfold and evolve over time, potentially aiding in more accurate emotion recognition. To address this, we propose RBTransformer, a Transformer-based neural network architecture that models inter-cortical neural dynamics of the brain in latent space to better capture structured neural interactions for effective EEG-based emotion recognition. First, the EEG signals are converted into Band Differential Entropy (BDE) tokens, which are then passed through Electrode Identity embeddings to retain spatial provenance. These tokens are processed through successive inter-cortical multi-head attention blocks that construct an electrode x electrode attention matrix, allowing the model to learn the inter-cortical neural dependencies. The resulting features are then passed through a classification head to obtain the final prediction. We conducted extensive experiments, specifically under subject-dependent settings, on the SEED, DEAP, and DREAMER datasets, over all three dimensions, Valence, Arousal, and Dominance (for DEAP and DREAMER), under both binary and multi-class classification settings. The results demonstrate that the proposed RBTransformer outperforms all previous state-of-the-art methods across all three datasets, over all three dimensions under both classification settings. The source code is available at: https://github.com/nnilayy/RBTransformer.

Figures (8)

• Figure 1: Visual representation of a wide range of emotions in the three-dimensional Valence–Arousal–Dominance (VAD) space.
• Figure 2: Individual emotion representations within the three-dimensional Valence–Arousal–Dominance (VAD) space, showcasing the unique characteristics of each emotion.
• Figure 3: Preprocessing pipeline applied across EEG datasets for RBTransformer.
• Figure 4: Schematic architecture diagram of RBTransformer implementing inter-cortical attention.
• Figure 5: Electrode layouts for (a) SEED zheng2015investigating with 62 electrodes, (b) DEAP koelstra2012deap with 32 electrodes, and (c) DREAMER katsigiannis2018dreamer with 14 electrodes. Blue circles indicate active electrodes used in each dataset; Grey circles denote unused electrodes.