Pretraining Large Brain Language Model for Active BCI: Silent Speech

TL;DR

This work tackles silent speech decoding for active BCI by introducing the Large Brain Language Model (LBLM), a 22-million-parameter backbone pretrained on a large EEG silent-speech dataset. The authors propose Future Spectro-Temporal Prediction (FSTP), a self-supervised paradigm combining Masked Spectro-Temporal Prediction (MSTP) and Autoregressive Spectro-Temporal Prediction (ASTP) to learn temporally and spectrally rich EEG representations, followed by a spatio-temporal classifier for downstream tasks. On a dataset of over 120 hours from 12 subjects and a 24-word vocabulary, the approach achieves state-of-the-art cross-session performance, with average word-level accuracy around 39.6% and semantic-level accuracy around 42.8%, and with notable subject-specific improvements. The work also demonstrates the model’s ability to predict short-term future EEG waves, offering insights into brain dynamics and potential improvements in the reliability of real-time active BCI systems, while contributing a publicly available dataset for further research.

Abstract

This paper explores silent speech decoding in active brain-computer interface (BCI) systems, which offer more natural and flexible communication than traditional BCI applications. We collected a new silent speech dataset of over 120 hours of electroencephalogram (EEG) recordings from 12 subjects, capturing 24 commonly used English words for language model pretraining and decoding. Following the recent success of pretraining large models with self-supervised paradigms to enhance EEG classification performance, we propose Large Brain Language Model (LBLM) pretrained to decode silent speech for active BCI. To pretrain LBLM, we propose Future Spectro-Temporal Prediction (FSTP) pretraining paradigm to learn effective representations from unlabeled EEG data. Unlike existing EEG pretraining methods that mainly follow a masked-reconstruction paradigm, our proposed FSTP method employs autoregressive modeling in temporal and frequency domains to capture both temporal and spectral dependencies from EEG signals. After pretraining, we finetune our LBLM on downstream tasks, including word-level and semantic-level classification. Extensive experiments demonstrate significant performance gains of the LBLM over fully-supervised and pretrained baseline models. For instance, in the difficult cross-session setting, our model achieves 47.0\% accuracy on semantic-level classification and 39.6\% in word-level classification, outperforming baseline methods by 5.4\% and 7.3\%, respectively. Our research advances silent speech decoding in active BCI systems, offering an innovative solution for EEG language model pretraining and a new dataset for fundamental research.

Figures (15)

• Figure 1: Overview of the proposed Future Spectro-Temporal Prediction (FSTP) pretraining paradigm for the LBLM backbone, which consists of two stages. In Stage (1), Masked Spectro-Temporal Prediction (MSTP) pretraining is used to warm up the model weights by reconstructing randomly masked EEG wave and spectra. In Stage (2), future EEG patches are completely masked out in Autoregressive Spectro-Temporal Prediction (ASTP) to help the model learn non-trivial representations. For classification, a spatio-temporal classifier is added to aggregate and select the representations learned by the LBLM backbone. We finetune the whole model for effective semantic-level and word-level classification tasks.
• Figure 2: The proposed LBLM backbone. EEG signals are first segmented into overlapping patches and embedded with positional and subject embeddings. We build the backbone model using Conformer blocks with a layer-gating mechanism to stabilize training. The gating connects the input and output tokens using a zero convolution layer with $1\times{1}$ convolution with both weight and bias initialied to zero.
• Figure 3: (a) In FSTP pretraining, the backbone model predicts the raw EEG wave, the Fourier amplitude and the Fourier phase from each output token of the LBLM backbone. A separate prediction head for each modality. (b) Sampling the input context and target window during ASTP pretraining.
• Figure 4: Detailed structure of the Spatio-Temporal (ST) classifier for downstream tasks exploiting EEG patterns learned from backbone LBLM. The ST classifier integrates features across EEG channels and extracts multi-scale temporal patterns to improve classification performance.
• Figure 5: EEG-based Silent Speech Experiment design. Each trial consisted of four segments: rest, read, preparation, and silent speech. The number in between indicates the duration of each segment.