Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EEG-to-Text Translation: A Model for Deciphering Human Brain Activity

Saydul Akbar Murad, Ashim Dahal, Nick Rahimi

TL;DR

This work tackles EEG-to-text translation by introducing R1 Translator, an encoder–decoder model that combines a Bi-LSTM EEG encoder with a pretrained BART text generator. It employs a two-step fine-tuning strategy to adapt the language model to EEG inputs, evaluated on ZuCo V1 and V2 datasets with strong, consistent improvements over T5 and Brain Translator across BLEU, ROUGE, WER, and CER metrics. The results establish state-of-the-art performance for EEG-to-text decoding on these datasets and demonstrate both robustness and practical potential, while also outlining limitations and directions for real-time, scalable brain-computer interfaces. Future work includes expanding the EEG corpus for broader generalization and integrating the system into real-time, decentralized architectures for instantaneous brain-to-text communication.

Abstract

With the rapid advancement of large language models like Gemini, GPT, and others, bridging the gap between the human brain and language processing has become an important area of focus. To address this challenge, researchers have developed various models to decode EEG signals into text. However, these models still face significant performance limitations. To overcome these shortcomings, we propose a new model, R1 Translator, which aims to improve the performance of EEG-to-text decoding. The R1 Translator model combines a bidirectional LSTM encoder with a pretrained transformer-based decoder, utilizing EEG features to produce high-quality text outputs. The model processes EEG embeddings through the LSTM to capture sequential dependencies, which are then fed into the transformer decoder for effective text generation. The R1 Translator excels in ROUGE metrics, outperforming both T5 (previous research) and Brain Translator. Specifically, R1 achieves a ROUGE-1 score of 38.00% (P), which is up to 9% higher than T5 (34.89%) and 3% better than Brain (35.69%). It also leads in ROUGE-L, with a F1 score of 32.51%, outperforming T5 by 3% (29.67%) and Brain by 2% (30.38%). In terms of CER, R1 achieves a CER of 0.5795, which is 2% lower than T5 (0.5917) and 4% lower than Brain (0.6001). Additionally, R1 performs better in WER with a score of 0.7280, outperforming T5 by 4.3% (0.7610) and Brain by 3.6% (0.7553). Code is available at https://github.com/Mmurrad/EEG-To-text.

EEG-to-Text Translation: A Model for Deciphering Human Brain Activity

TL;DR

This work tackles EEG-to-text translation by introducing R1 Translator, an encoder–decoder model that combines a Bi-LSTM EEG encoder with a pretrained BART text generator. It employs a two-step fine-tuning strategy to adapt the language model to EEG inputs, evaluated on ZuCo V1 and V2 datasets with strong, consistent improvements over T5 and Brain Translator across BLEU, ROUGE, WER, and CER metrics. The results establish state-of-the-art performance for EEG-to-text decoding on these datasets and demonstrate both robustness and practical potential, while also outlining limitations and directions for real-time, scalable brain-computer interfaces. Future work includes expanding the EEG corpus for broader generalization and integrating the system into real-time, decentralized architectures for instantaneous brain-to-text communication.

Abstract

With the rapid advancement of large language models like Gemini, GPT, and others, bridging the gap between the human brain and language processing has become an important area of focus. To address this challenge, researchers have developed various models to decode EEG signals into text. However, these models still face significant performance limitations. To overcome these shortcomings, we propose a new model, R1 Translator, which aims to improve the performance of EEG-to-text decoding. The R1 Translator model combines a bidirectional LSTM encoder with a pretrained transformer-based decoder, utilizing EEG features to produce high-quality text outputs. The model processes EEG embeddings through the LSTM to capture sequential dependencies, which are then fed into the transformer decoder for effective text generation. The R1 Translator excels in ROUGE metrics, outperforming both T5 (previous research) and Brain Translator. Specifically, R1 achieves a ROUGE-1 score of 38.00% (P), which is up to 9% higher than T5 (34.89%) and 3% better than Brain (35.69%). It also leads in ROUGE-L, with a F1 score of 32.51%, outperforming T5 by 3% (29.67%) and Brain by 2% (30.38%). In terms of CER, R1 achieves a CER of 0.5795, which is 2% lower than T5 (0.5917) and 4% lower than Brain (0.6001). Additionally, R1 performs better in WER with a score of 0.7280, outperforming T5 by 4.3% (0.7610) and Brain by 3.6% (0.7553). Code is available at https://github.com/Mmurrad/EEG-To-text.

Paper Structure

This paper contains 26 sections, 18 equations, 4 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. ZuCo datasets
  5. R1 Translator Architecture
  6. EEG Encoder
  7. Linear Projection
  8. BART-based Text Generation
  9. BART Encoder
  10. BART Decoder
  11. Two-Step Fine-Tuning Strategy
  12. Step 1: Partial Training and Feature Alignment
  13. Step 2: End-to-End Fine-Tuning
  14. Optimization
  15. Implementation Details
  16. ...and 11 more sections

Figures (4)

  • Figure 1: R1Translator model architecture and training principle.
  • Figure 2: Evaluation of SacreBLEU, Word Error Rate (WER), and Character Error Rate (CER) with and without teaching force (w/tf) for EEG-to-text decoding on the ZuCo dataset, across Reading Task versions SR V1, NR V1, and TSR V1.
  • Figure 3: Evaluation of SacreBLEU, Word Error Rate (WER), and Character Error Rate (CER) with and without teaching force (w/tf) for EEG-to-text decoding on the ZuCo dataset, across Reading Task versions NR V2 and TSR V2.
  • Figure 4: Evaluation of SacreBLEU, Word Error Rate (WER), and Character Error Rate (CER) with and without teaching force (w/tf) for EEG-to-text decoding on the ZuCo dataset, across Reading Task versions SR V1.0, NR V1.0, TSR V1, and TSR V2.