Affect and Effect: Limitations of regularisation-based continual learning in EEG-based emotion classification

TL;DR

This work shows that regularisation-based continual learning methods for EEG-based emotion classification struggle to generalise to unseen subjects due to a misalignment between stability-focused objectives and forward transfer needs. Through theoretical analysis and empirical evaluation on DREAMER and SEED, it shows that importance-heuristic estimators are fragile under EEG noise, gradient updates for new subjects interfere with past knowledge, and accumulated importance over tasks leads to early network freezing. Forward transfer is not significantly better than sequential fine-tuning, while backward transfer can improve past-task stability at the expense of plasticity, highlighting a fundamental limitation for subject-incremental learning in this domain. The findings motivate exploring meta-learning and EEG foundation models to achieve robust cross-subject generalisation in real-world BCI/affect applications.

Abstract

Generalisation to unseen subjects in EEG-based emotion classification remains a challenge due to high inter-and intra-subject variability. Continual learning (CL) poses a promising solution by learning from a sequence of tasks while mitigating catastrophic forgetting. Regularisation-based CL approaches, such as Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), and Memory Aware Synapses (MAS), are commonly used as baselines in EEG-based CL studies, yet their suitability for this problem remains underexplored. This study theoretically and empirically finds that regularisation-based CL methods show limited performance for EEG-based emotion classification on the DREAMER and SEED datasets. We identify a fundamental misalignment in the stability-plasticity trade-off, where regularisation-based methods prioritise mitigating catastrophic forgetting (backward transfer) over adapting to new subjects (forward transfer). We investigate this limitation under subject-incremental sequences and observe that: (1) the heuristics for estimating parameter importance become less reliable under noisy data and covariate shift, (2) gradients on parameters deemed important by these heuristics often interfere with gradient updates required for new subjects, moving optimisation away from the minimum, (3) importance values accumulated across tasks over-constrain the model, and (4) performance is sensitive to subject order. Forward transfer showed no statistically significant improvement over sequential fine-tuning (p > 0.05 across approaches and datasets). The high variability of EEG signals means past subjects provide limited value to future subjects. Regularisation-based continual learning approaches are therefore limited for robust generalisation to unseen subjects in EEG-based emotion classification.

Figures (27)

• Figure 1: EEGNet Architecture Details as adapted from lawhern_eegnet_2018
• Figure 9: Effect of increasing the sample size on convergence of empirical Fisher Information to the true Fisher Information.
• Figure 10: Effect of decreasing the batch size on inflation of the path integral in synaptic intelligence.
• Figure 11: Effect of decreasing the batch size on magnitude of importance estimates in memory aware synapses.
• Figure 12: [Naïve] Cosine similarity of all gradients between subsequent tasks without regularisation. The $y$-axis is fixed to the full cosine similarity range $[-1, 1]$ to preserve interpretability and comparability across task pairs and methods. No per-bar parameter-count labels are shown, as all parameters are shared between tasks (100% overlap) for the naïve strategy.