Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Cognition Model: Towards Pretrained EEG Foundation Model

Chi-Sheng Chen, Ying-Jung Chen, Aidan Hung-Wen Tsai

TL;DR

EEG data suffer from low SNR and high inter-subject variability, limiting scalable foundation models. The authors propose Large Cognition Model (LCM), a transformer-based EEG foundation model trained with self-supervised contrastive learning and cross-montage encoding to enable generalization across datasets and tasks. Key contributions include a learnable cross-montage channel mapping, a dual-encoder contrastive framework with masked feature reconstruction, and strong cross-dataset performance that surpasses several pretrained baselines even without extensive pretraining. This work advances universal EEG representations with potential impact on neuroscience, personalized medicine, and brain-computer interface technologies.

Abstract

Electroencephalography provides a non-invasive window into brain activity, offering valuable insights for neurological research, brain-computer interfaces, and clinical diagnostics. However, the development of robust machine learning models for EEG analysis is hindered by the scarcity of large-scale, well-annotated datasets and the inherent variability of EEG signals across subjects and recording conditions. Inspired by the success of foundation models in natural language processing and computer vision, we propose the Large Cognition Model-a transformer-based foundation model designed to generalize across diverse EEG datasets and downstream tasks. Unlike traditional approaches, our proposed transformer-based architecture demonstrates strong generalization capabilities across datasets and tasks, even without pretraining, surpassing some existing EEG universal models on specific downstream applications. LCM leverages large-scale self-supervised learning techniques to capture universal EEG representations, enabling efficient fine-tuning for applications such as cognitive state decoding, disease classification, and neurofeedback systems. We introduce a novel architecture that integrates temporal and spectral attention mechanisms, optimizing the model's ability to extract meaningful features from raw EEG signals. Extensive evaluations demonstrate that LCM outperforms state-of-the-art approaches across multiple EEG benchmarks, exhibiting strong cross-subject and cross-task generalization. Our findings highlight the potential of pretrained EEG foundation models to accelerate advancements in neuroscience, personalized medicine, and BCI technology.

Large Cognition Model: Towards Pretrained EEG Foundation Model

TL;DR

EEG data suffer from low SNR and high inter-subject variability, limiting scalable foundation models. The authors propose Large Cognition Model (LCM), a transformer-based EEG foundation model trained with self-supervised contrastive learning and cross-montage encoding to enable generalization across datasets and tasks. Key contributions include a learnable cross-montage channel mapping, a dual-encoder contrastive framework with masked feature reconstruction, and strong cross-dataset performance that surpasses several pretrained baselines even without extensive pretraining. This work advances universal EEG representations with potential impact on neuroscience, personalized medicine, and brain-computer interface technologies.

Abstract

Electroencephalography provides a non-invasive window into brain activity, offering valuable insights for neurological research, brain-computer interfaces, and clinical diagnostics. However, the development of robust machine learning models for EEG analysis is hindered by the scarcity of large-scale, well-annotated datasets and the inherent variability of EEG signals across subjects and recording conditions. Inspired by the success of foundation models in natural language processing and computer vision, we propose the Large Cognition Model-a transformer-based foundation model designed to generalize across diverse EEG datasets and downstream tasks. Unlike traditional approaches, our proposed transformer-based architecture demonstrates strong generalization capabilities across datasets and tasks, even without pretraining, surpassing some existing EEG universal models on specific downstream applications. LCM leverages large-scale self-supervised learning techniques to capture universal EEG representations, enabling efficient fine-tuning for applications such as cognitive state decoding, disease classification, and neurofeedback systems. We introduce a novel architecture that integrates temporal and spectral attention mechanisms, optimizing the model's ability to extract meaningful features from raw EEG signals. Extensive evaluations demonstrate that LCM outperforms state-of-the-art approaches across multiple EEG benchmarks, exhibiting strong cross-subject and cross-task generalization. Our findings highlight the potential of pretrained EEG foundation models to accelerate advancements in neuroscience, personalized medicine, and BCI technology.

Paper Structure

This paper contains 13 sections, 14 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Feature Extraction and Cross-Montage Encoding
  4. Spatio-Temporal Contrastive Alignment
  5. Mask-Based Reconstruction
  6. Optimization and Learning Rate Scheduling
  7. Training Process and Gradient Updates
  8. Gradient Logging and Adaptation
  9. Experiments and Results
  10. Dataset and Preprocessing
  11. Experiment Details
  12. Results
  13. Conclusions

Figures (1)

  • Figure 1: Overview of the LCM Training Flow. The EEG signals are first segmented into spatio-temporal patches. Each patch is then assigned a channel embedding to encode electrode-specific information. A subset of the patches is masked for self-supervised learning. The masked and unmasked patches are processed through the Latent Contrastive Masking (LCM) module, which consists of a convolutional block followed by transformer layers to extract hierarchical spatio-temporal representations. Finally, contrastive learning is applied to align learned EEG representations, improving the robustness and generalizability of the model.