Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

Zhenhailong Wang, Heng Ji

TL;DR

The paper tackles open vocabulary brain–computer interfacing by decoding English text from noninvasive EEG signals using a pretrained language model (BART) as a core component, treating the brain as a text encoder. It introduces open vocabulary EEG-To-Text decoding and zero-shot EEG-based sentence sentiment classification, achieving a BLEU-1 of 40.1% and a zero-shot sentiment F1 of 55.6% on natural reading tasks. The approach maps EEG features to a latent space compatible with BART via an additional Transformer encoder, enabling generation from a large vocabulary ($| abla\mathcal{V}|\sim 50{,}000$) and transferable sentiment analysis without EEG–sentiment labels. The results demonstrate cross-subject robustness, improved performance with more diverse training data, and strong potential for noninvasive, high-coverage brain-to-text systems, with future work on larger datasets and multilingual inner speech decoding.

Abstract

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. In addition, most of the high-performing approaches require data from invasive devices (e.g., ECoG). In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. We hypothesis that the human brain functions as a special text encoder and propose a novel framework leveraging pre-trained language models (e.g., BART). Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for a high-performance open vocabulary brain-to-text system once sufficient data is available

Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

TL;DR

The paper tackles open vocabulary brain–computer interfacing by decoding English text from noninvasive EEG signals using a pretrained language model (BART) as a core component, treating the brain as a text encoder. It introduces open vocabulary EEG-To-Text decoding and zero-shot EEG-based sentence sentiment classification, achieving a BLEU-1 of 40.1% and a zero-shot sentiment F1 of 55.6% on natural reading tasks. The approach maps EEG features to a latent space compatible with BART via an additional Transformer encoder, enabling generation from a large vocabulary () and transferable sentiment analysis without EEG–sentiment labels. The results demonstrate cross-subject robustness, improved performance with more diverse training data, and strong potential for noninvasive, high-coverage brain-to-text systems, with future work on larger datasets and multilingual inner speech decoding.

Abstract

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. In addition, most of the high-performing approaches require data from invasive devices (e.g., ECoG). In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. We hypothesis that the human brain functions as a special text encoder and propose a novel framework leveraging pre-trained language models (e.g., BART). Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for a high-performance open vocabulary brain-to-text system once sufficient data is available
Paper Structure (21 sections, 3 equations, 2 figures, 7 tables)

This paper contains 21 sections, 3 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Open vocabulary EEG-To-Text decoding
  4. EEG-based sentence sentiment classification
  5. Method
  6. EEG-To-Text Decoding
  7. Zero-shot Sentiment Classification Pipeline
  8. Experiment
  9. Dataset
  10. EEG-To-Text Decoding Evaluation
  11. Results
  12. Discussion
  13. EEG-based Sentiment Classification Evaluation
  14. Baselines
  15. Pipeline
  16. ...and 6 more sections

Figures (2)

  • Figure 1: An overview of proposed tasks. Subjects are asked to read text on a screen at their own speed. Simultaneous eye-tracking and EEG data are recorded. The darker the background color, the more fixations are on the word. Word-level EEG features can be extracted by synchronizing with eye-tracking fixations. EEG feature sequences then serve as inputs for sequence-to-sequence decoding or sentiment classification. In this paper, we use ZuCo datasets for experiments, please refer to Section \ref{['subsec:experiment_dataset']} for more details.
  • Figure 2: (a) Our proposed framework for EEG-To-Text decoding. (b) Our proposed pipeline for zero-shot EEG-based sentence sentiment classification. EEG-To-Text model trained on EEG-Sentence pairs in (a) is plugged in as Decoder in (b). Sequence classification model trained on external Text-Sentiment pairs is plugged in as Classifier in (b).