Robust Emotion Recognition via Bi-Level Self-Supervised Continual Learning

TL;DR

This work tackles EEG-based emotion recognition under realistic streaming conditions where labels are unavailable and cross-subject variability is high. It introduces SSOCL, a bi-level self-supervised continual learning framework that maintains a compact, dynamically updated memory buffer and uses a future-prediction self-supervised objective plus a cluster-mapping mechanism to assign pseudo-labels. An outer memory-enhancement module selects reliable, low-entropy samples for replay, while a meta-loop trains on both current data and memory, enabling robust cross-subject generalization without labels. Experiments on AMIGOS, DEAP, and PPB-EMO show SSOCL outperforming supervised and SSL baselines, with ablations confirming the importance of temporal self-supervision and entropy-based memory selection for reducing forgetting in streaming EEG data.

Abstract

Emotion recognition through physiological signals such as electroencephalogram (EEG) has become an essential aspect of affective computing and provides an objective way to capture human emotions. However, physiological data characterized by cross-subject variability and noisy labels hinder the performance of emotion recognition models. Existing domain adaptation and continual learning methods struggle to address these issues, especially under realistic conditions where data is continuously streamed and unlabeled. To overcome these limitations, we propose a novel bi-level self-supervised continual learning framework, SSOCL, based on a dynamic memory buffer. This bi-level architecture iteratively refines the dynamic buffer and pseudo-label assignments to effectively retain representative samples, enabling generalization from continuous, unlabeled physiological data streams for emotion recognition. The assigned pseudo-labels are subsequently leveraged for accurate emotion prediction. Key components of the framework, including a fast adaptation module and a cluster-mapping module, enable robust learning and effective handling of evolving data streams. Experimental validation on two mainstream EEG tasks demonstrates the framework's ability to adapt to continuous data streams while maintaining strong generalization across subjects, outperforming existing approaches.

Figures (4)

• Figure 1: SSOCL framework: Inner-loop: In a time step $t$, the feature extractor adapts to the current data $D^t = {x^t}$ in a self-supervised manner by using a self supervised module (SSM), which consists of a predictor network that takes current embeddings $z_1^t$ and predicts future embeddings $\hat{z}_2^t$. After adaptation, k-means clustering partitions $\bm{Z}^t$ into $k$ clusters. To assign the pseudo labels, the cluster mapping module maps the newly formed clusters to the existing data stored in a memory buffer. Meta-loop: The current embeddings and the memory data embeddings are combined for emotion prediction. Finally, a memory enhancement module (MEnM) refines the samples for replay in the next time step $t+1$.
• Figure 2: The memory buffer obtained using the SSOCL approach at the end of training on the DEAP dataset. The embeddings are generated using the model trained with SSOCL, and UMAP is applied to visualize these embeddings in a 2-dimensional plane. Colors represent the pseudo labels assigned to the stored samples.
• Figure 3: Visualization of the embedding spaces learned by SSOCL (top rows: (a–f)) and SCALE (bottom rows: (g–l)). The feature extractor trained by each method on the AMIGOS dataset in OCL settings is used to compute embedding vectors for the test datasets of a few subjects. These embeddings are visualized with UMAP in a 2D plane. Each panel represents the embedding space for a particular subject, where each point corresponds to a test sample and the colors indicate the true labels.
• Figure 4: UMAP visualizations of memory buffer embeddings for different temperature values: (a) $T = 1$, (b) $T = 10$, (c) $T = 100$, and (d) $T = 1000$. At $T = 1$ and $T = 10$, intra-class embeddings are highly clustered with minimal diversity. At $T = 100$, clear and well-separated class boundaries emerge. At $T = 1000$, the embeddings become blurred, with significant overlap between clusters, indicating a loss of distinct boundaries.