HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition

TL;DR

HADUA targets robust cross-subject emotion recognition from multimodal EEG and eye-movement signals by integrating hierarchical attention-based fusion with confidence-aware pseudo-label weighting and uniform class-level alignment. The method jointly learns discriminative, modality-aware representations and performs both marginal and conditional distribution alignment across subjects, using target pseudo-labels refined through Soft Gaussian Weighting and Uniform Alignment. Empirical results on SEED, SEED-IV, and related datasets show state-of-the-art accuracy and balanced class performance, with ablations confirming the complementary benefits of each component. This approach promises practical robustness for cross-subject affective computing in real-world HCI and brain-computer interface systems.

Abstract

Robust cross-subject emotion recognition from multimodal physiological signals remains a challenging problem, primarily due to modality heterogeneity and inter-subject distribution shift. To tackle these challenges, we propose a novel adaptive learning framework named Hierarchical Attention and Dynamic Uniform Alignment (HADUA). Our approach unifies the learning of multimodal representations with domain adaptation. First, we design a hierarchical attention module that explicitly models intra-modal temporal dynamics and inter-modal semantic interactions (e.g., between electroencephalogram(EEG) and eye movement(EM)), yielding discriminative and semantically coherent fused features. Second, to overcome the noise inherent in pseudo-labels during adaptation, we introduce a confidence-aware Gaussian weighting scheme that smooths the supervision from target-domain samples by down-weighting uncertain instances. Third, a uniform alignment loss is employed to regularize the distribution of pseudo-labels across classes, thereby mitigating imbalance and stabilizing conditional distribution matching. Extensive experiments on multiple cross-subject emotion recognition benchmarks show that HADUA consistently surpasses existing state-of-the-art methods in both accuracy and robustness, validating its effectiveness in handling modality gaps, noisy pseudo-labels, and class imbalance. Taken together, these contributions offer a practical and generalizable solution for building robust cross-subject affective computing systems.

Figures (9)

• Figure 1: Overview of the proposed cross-subject multimodal emotion recognition framework. The framework mainly consists of three mutually collaborative components: (1) Attention-based multimodal feature fusion module, which employs modality-specific self-attention and cross-modal attention mechanisms to jointly model the temporal structures of EEG and eye-movement signals as well as their cross-modal semantic relationships, yielding more discriminative fused representations; (2) Multi-level distribution alignment module, which aligns the marginal distributions between the source and target domains using MMD, and further performs conditional distribution alignment via CMMD based on target-domain pseudo-labels to alleviate cross-subject domain shift; (3) Confidence-driven pseudo-label optimization module, which applies confidence-based Gaussian modeling (CDE), class distribution estimation (CGM), and a uniform distribution alignment strategy to perform sample-level weighting of target-domain pseudo-labels, thereby suppressing the adverse effects of low-reliability pseudo-labels during model optimization.
• Figure 2: Evolution of the truncated soft Gaussian weighting function. Samples with prediction confidence above the adaptive threshold $\mu_t$ receive the full weight $\lambda_{\max}$, while those below $\mu_t$ are softly down-weighted following a Gaussian decay.
• Figure 3: Confusion matrices of HADUA on (a) SEED and (b) SEED-IV datasets. For SEED, the emotion categories are Negative, Neutral, and Positive. For SEED-IV, the categories are neutral, sad, fear, and happy.
• Figure 4: Visualization of feature evolution on the SEED and SEED-IV datasets. (a–c) show the feature distributions of SEED at epochs 0, 50, and 200, respectively; (d–f) present the feature distributions of SEED-IV at the corresponding epochs. Different colors denote different emotion categories, while circular and star markers represent source-domain and target-domain samples, respectively.
• Figure 5: Feature importance (Mutual Information) on SEED. Frontal electrodes (F5, Fpz) exhibit the highest discriminative power, peaking in the Gamma band.