Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

Yuqi Chen, Kan Ren, Kaitao Song, Yansen Wang, Yifan Wang, Dongsheng Li, Lili Qiu

TL;DR

EEGFormer tackles the challenge of learning transferable, interpretable representations from large-scale unlabeled EEG data. It introduces a discrete vector-quantized pretraining framework that tokenizes multivariate EEG patches via a codebook and trains a lightweight decoder to reconstruct inputs. Pretraining on 1.7TB TUH data yields universal representations that transfer to multiple downstream tasks, including seizure detection and neonatal seizure localization, with improved metrics over baselines. The model supports interpretability by analyzing discrete tokens and codebook indices, enabling localization of salient patterns without heavy supervision.

Abstract

Self-supervised learning has emerged as a highly effective approach in the fields of natural language processing and computer vision. It is also applicable to brain signals such as electroencephalography (EEG) data, given the abundance of available unlabeled data that exist in a wide spectrum of real-world medical applications ranging from seizure detection to wave analysis. The existing works leveraging self-supervised learning on EEG modeling mainly focus on pretraining upon each individual dataset corresponding to a single downstream task, which cannot leverage the power of abundant data, and they may derive sub-optimal solutions with a lack of generalization. Moreover, these methods rely on end-to-end model learning which is not easy for humans to understand. In this paper, we present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. The pretrained model cannot only learn universal representations on EEG signals with adaptable performance on various downstream tasks but also provide interpretable outcomes of the useful patterns within the data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings. Furthermore, we demonstrate how the learned model exhibits transferable anomaly detection performance and provides valuable interpretability of the acquired patterns via self-supervised learning.

EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Model

TL;DR

EEGFormer tackles the challenge of learning transferable, interpretable representations from large-scale unlabeled EEG data. It introduces a discrete vector-quantized pretraining framework that tokenizes multivariate EEG patches via a codebook and trains a lightweight decoder to reconstruct inputs. Pretraining on 1.7TB TUH data yields universal representations that transfer to multiple downstream tasks, including seizure detection and neonatal seizure localization, with improved metrics over baselines. The model supports interpretability by analyzing discrete tokens and codebook indices, enabling localization of salient patterns without heavy supervision.

Abstract

Self-supervised learning has emerged as a highly effective approach in the fields of natural language processing and computer vision. It is also applicable to brain signals such as electroencephalography (EEG) data, given the abundance of available unlabeled data that exist in a wide spectrum of real-world medical applications ranging from seizure detection to wave analysis. The existing works leveraging self-supervised learning on EEG modeling mainly focus on pretraining upon each individual dataset corresponding to a single downstream task, which cannot leverage the power of abundant data, and they may derive sub-optimal solutions with a lack of generalization. Moreover, these methods rely on end-to-end model learning which is not easy for humans to understand. In this paper, we present a novel EEG foundation model, namely EEGFormer, pretrained on large-scale compound EEG data. The pretrained model cannot only learn universal representations on EEG signals with adaptable performance on various downstream tasks but also provide interpretable outcomes of the useful patterns within the data. To validate the effectiveness of our model, we extensively evaluate it on various downstream tasks and assess the performance under different transfer settings. Furthermore, we demonstrate how the learned model exhibits transferable anomaly detection performance and provides valuable interpretability of the acquired patterns via self-supervised learning.
Paper Structure (21 sections, 1 equation, 3 figures, 2 tables)

This paper contains 21 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pretraining for Time-Series Data
  4. Pretraining for EEG data
  5. EEGFormer: Vector-Quantized Pretraining Transformer for EEG Data
  6. Feature Preprocessing
  7. Slice & Encode
  8. Vector Quantizer
  9. Decode & Reconstruct
  10. Training Loss
  11. Downstream Fine-Tuning
  12. Experimental Results
  13. Datasets Description
  14. Parameter Setting
  15. Compared Baselines
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Overview of EEGFormer. Initially, multi-variate EEG signals are segmented into patches, which are then passed through a Transformer encoder. Subsequently, a vector-quantized model is employed to generate discrete indices. These indices are then fed into a shallow Transformer decoder.
  • Figure 2: Influence of pretrain epochs on two TUH corpus.
  • Figure 3: Interpretation results from naive Bayes model.