Thought2Text: Text Generation from EEG Signal using Large Language Models (LLMs)

TL;DR

This work addresses translating EEG-brain activity into natural language by aligning EEG embeddings with vision-language representations and fine-tuning instruction-tuned LLMs. It introduces Thought2Text, a three-stage pipeline (EEG encoder training, image-embedding priming, and EEG-embedding tuning) that enables text generation from EEG with no image input at inference. Evaluation on a public CVPR2017 EEG dataset across six subjects demonstrates improvements over baselines and robustness across subjects, using both traditional metrics and GPT-4-based fluency/adequacy assessments. The approach holds promise for accessible thought-to-text systems in neuroscience and NLP, while acknowledging EEG noise, misclassification, and privacy considerations as areas for future work.

Abstract

Decoding and expressing brain activity in a comprehensible form is a challenging frontier in AI. This paper presents Thought2Text, which uses instruction-tuned Large Language Models (LLMs) fine-tuned with EEG data to achieve this goal. The approach involves three stages: (1) training an EEG encoder for visual feature extraction, (2) fine-tuning LLMs on image and text data, enabling multimodal description generation, and (3) further fine-tuning on EEG embeddings to generate text directly from EEG during inference. Experiments on a public EEG dataset collected for six subjects with image stimuli and text captions demonstrate the efficacy of multimodal LLMs (LLaMA-v3, Mistral-v0.3, Qwen2.5), validated using traditional language generation evaluation metrics, as well as fluency and adequacy measures. This approach marks a significant advancement towards portable, low-cost "thoughts-to-text" technology with potential applications in both neuroscience and natural language processing.

Figures (3)

• Figure 1: Multi-stage training process for Thought2Text. Stage 1: EEG ChannelNet encoder is trained using MSE loss (aligning EEG embeddings with CLIP model image embeddings) and CE loss (for EEG classification). Stage 2: LLMs are fine-tuned using image descriptions and object labels, with only the projector trained while the LLM and CLIP encoder remain frozen. Stage 3: Similar to stage 2, but the projector is trained with EEG embeddings. LLM and EEG encoder remain frozen.
• Figure 2: Subjectwise analysis comparing $\mathrm{ALL}$ and $\mathrm{NO\_STAGE2}$ variants across subjects for GPT-4 Adequacy and BLEU Unigram metrics, evaluated using different models. For BLEU scores, the $\mathrm{ALL}$ variants show a noticeable improvement across all six subjects compared to the $\mathrm{NO\_STAGE2}$ variants. Although numerically smaller, a consistent improvement in Adequacy is also observed with the $\mathrm{ALL}$ variants, which is significant in context of noisy EEG data.
• Figure 3: Confusion Matrix for classification results from stage 1. Some misclassifications are flowers being identified as mushrooms, airplane as identified as clownfish or fish, elephant as a panda etc.