Neuro-BERT: Rethinking Masked Autoencoding for Self-supervised Neurological Pretraining

TL;DR

Neuro-BERT tackles data scarcity in neurological signal analysis by introducing Fourier Inversion Prediction within a masked autoencoding framework. It uses 1-D patch embeddings and a Transformer encoder to learn representations without heavy data augmentation, predicting Fourier-domain magnitude and phase, and reconstructing via inverse transform. Across epilepsy, sleep staging, and EMG tasks, Neuro-BERT achieves state-of-the-art fine-tuning and robust semi-supervised and transfer performance, often outperforming contrastive and traditional MAE baselines. This approach offers a data-efficient, interpretable pretraining paradigm for EEG/EMG that can benefit downstream neurological diagnostics and brain-computer interface applications.

Abstract

Deep learning associated with neurological signals is poised to drive major advancements in diverse fields such as medical diagnostics, neurorehabilitation, and brain-computer interfaces. The challenge in harnessing the full potential of these signals lies in the dependency on extensive, high-quality annotated data, which is often scarce and expensive to acquire, requiring specialized infrastructure and domain expertise. To address the appetite for data in deep learning, we present Neuro-BERT, a self-supervised pre-training framework of neurological signals based on masked autoencoding in the Fourier domain. The intuition behind our approach is simple: frequency and phase distribution of neurological signals can reveal intricate neurological activities. We propose a novel pre-training task dubbed Fourier Inversion Prediction (FIP), which randomly masks out a portion of the input signal and then predicts the missing information using the Fourier inversion theorem. Pre-trained models can be potentially used for various downstream tasks such as sleep stage classification and gesture recognition. Unlike contrastive-based methods, which strongly rely on carefully hand-crafted augmentations and siamese structure, our approach works reasonably well with a simple transformer encoder with no augmentation requirements. By evaluating our method on several benchmark datasets, we show that Neuro-BERT improves downstream neurological-related tasks by a large margin.

Figures (6)

• Figure 1: Example of the proposed Fourier inversion prediction on EMG signal. The model predicts the phase and amplitude of the Fourier spectrum and reconstructs the missing signal segments using the inverse Fourier transform (IFT). For each masked signal segment, we illustrate the predicted phase and amplitude components of the Fourier spectrum as well as the corresponding original and reconstructed signal.
• Figure 2: Illustration of Neuro-BERT for neurological self-supervised pre-training. The input neurological signal is segmented into non-overlapping frames and transformed into a sequence of patch embeddings $e$ using a 1-D convolutional layer, where $N$ denotes the signal length and $k$ the convolution kernel size. Each patch embedding is added with a learnable positional embedding $p$ before being processed by the transformer encoder. The encoder output $r$ serves as the neurological patch representation. During self-supervised pre-training, a subset of these patches is randomly masked—represented by $[M]$ for the learnable mask token—and the model is trained to predict their frequency and phase distribution. The original signal is then reconstructed using the inverse discrete Fourier transform (IDFT). The Fourier prediction head is discarded after pre-training, and a linear prediction head will be adopted for various classification or regression downstream tasks on top of the pre-trained model.
• Figure 3: Visualization of pre-trained and fine-tuned embeddings with UMAP of hand gesture classification task on the Ninapro dataset. The first row shows the visualization of embeddings after pre-training, while the second row shows the visualization of embeddings after fine-tuning. Contrastive learning-based methods demonstrate better discrimination ability after pre-training than MaskedAE and our proposed method. However, fine-tuning contrastive-based methods brings limited improvement, while masked autoencoding-based methods such as MaskedAE and our proposed methods gain huge improvement from fine-tuning.
• Figure 4: Top-1 accuracy (%) of fine-tuning protocol for the semi-supervised scenario on SleepEDF and Ninapro datasets.
• Figure 5: Transfer learning performance comparison of pre-training between the classification and regression subsets of Ninapro datasets. Original indicates the performance of fine-tuning on the pre-trained weights with the targeted task labels directly without transferring.