Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings

TL;DR

This work tackles the challenge of decoding linguistic content from non-invasive EEG by identifying limitations of end-to-end LLM-based EEG-to-text systems, notably encoder semantics and decoder memorization. It introduces EEG-to-Text Retrieval (ETER), a two-step approach that first learns word-level semantic EEG representations with a Conformer encoder trained via masked contrastive learning, and then retrieves sentences via a training-free beam-search mechanism using keyword sets derived from the EEG predictions. The method is validated on the ZuCo corpus, showing meaningful word-level semantic organization and effective sentence retrieval without relying on a generative decoder, with extensive ablations exploring encoder architecture and vocabulary size. This retrieval-based framework offers a robust direction for linguistic BCIs, enabling transparent evaluation of semantic learning and scalable sentence decoding, while highlighting practical challenges in vocabulary scalability and dataset size.

Abstract

Decoding linguistic information from non-invasive brain signals using EEG has gained increasing research attention due to its vast applicational potential. Recently, a number of works have adopted a generative-based framework to decode electroencephalogram (EEG) signals into sentences by utilizing the power generative capacity of pretrained large language models (LLMs). However, this approach has several drawbacks that hinder the further development of linguistic applications for brain-computer interfaces (BCIs). Specifically, the ability of the EEG encoder to learn semantic information from EEG data remains questionable, and the LLM decoder's tendency to generate sentences based on its training memory can be hard to avoid. These issues necessitate a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Extensive experiments and ablation studies were conducted in this paper for a comprehensive evaluation of the proposed approach. Visualization of the top prediction candidates reveals that our model effectively groups EEG segments into semantic categories with similar meanings, thereby validating its ability to learn patterns from unspoken EEG recordings. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.

Figures (10)

• Figure 1: The overall structure of EEG-to-text retrieval approach. Our two-step approach consists of an word-level EEG encoder that encodes and prediction word-level results from EEG signals. Subsequently, a retrival method is applied to find the most relevant sentence utlizing the word-level results.
• Figure 2: After segmenting and tokenizing EEG signals at the word level, an EEG encoder comprising two Conformer blocks learns semantic patterns from the EEG signals. We implement masked contrastive learning, leveraging a frozen language model to provide language supervision to the EEG representations. The EEG encoder predicts a keyword set independently for each input EEG segment. Finally, a sentence retriever utilizes these predicted keyword sets to identify the most pertinent sentence from the dataset corpus.
• Figure 3: The architecture of the proposed EEG encoder. We first tokenize the low-level representations of EEG segments using a spatial encoder on the channel dimension and add positional embedding to indicate the frequency range of the token. Then we use $2$ conformer blocks to further process these tokens. The Conformer block encodes input tokens using a multi-head self-attention (MHSA) and a convolution module. For classification, we use an adaptive max pooling layer to aggregate the output of all EEG tokens into the final EEG representation $\mathbf{h}$ for each word. For classification, a fully-connected layer will be used to the classification distribution of the input EEG signal.
• Figure 4: The detailed structure of the convolution module used in the Conformer blocks.
• Figure 5: Our masked contrastive training scheme. Following CLIP radford2021learning and FLIP li2023scaling, we perform contrastive learning on pairs of EEG segments and words. We randomly mask out EEG tokens with a small masking ratio and encode only the visible EEG tokens. Here, the slash-shadowed blocks mean the EEG token is masked and is not visible to the EEG encoder. We do not mask word tokens in our paper.