Neural Spelling: A Spell-Based BCI System for Language Neural Decoding

TL;DR

This work addresses the lack of complete alphabet coverage in non-invasive BCI language decoding by introducing a Curriculum-based Neural Spelling (CNS) framework. CNS decouples EEG-to-letter decoding (via a CNN-based encoder and a neural-letter classifier) from sentence generation by leveraging curriculum-learning to fine-tune a pretrained large language model, translating noisy letter streams into fluent text. The two-stage approach yields strong stage-1 letter accuracy and notable stage-2 sentence-generation performance, demonstrated on EEG handwriting data and a creative-story corpus, with ablations clarifying the role of sampling and spacing. The combination of non-invasive EEG with GenAI enables scalable, inclusive communication solutions, while acknowledging limitations in online validation and cross-subject generalization that warrant future work.

Abstract

Brain-computer interfaces (BCIs) present a promising avenue by translating neural activity directly into text, eliminating the need for physical actions. However, existing non-invasive BCI systems have not successfully covered the entire alphabet, limiting their practicality. In this paper, we propose a novel non-invasive EEG-based BCI system with Curriculum-based Neural Spelling Framework, which recognizes all 26 alphabet letters by decoding neural signals associated with handwriting first, and then apply a Generative AI (GenAI) to enhance spell-based neural language decoding tasks. Our approach combines the ease of handwriting with the accessibility of EEG technology, utilizing advanced neural decoding algorithms and pre-trained large language models (LLMs) to translate EEG patterns into text with high accuracy. This system show how GenAI can improve the performance of typical spelling-based neural language decoding task, and addresses the limitations of previous methods, offering a scalable and user-friendly solution for individuals with communication impairments, thereby enhancing inclusive communication options.

Figures (8)

• Figure 1: Demonstrations of two typical language decoding frameworks. (A) Direct speech synthesis approach. (B) Spelling-based approach using phonemes. (C) Direct text synthesis approach. (D) Spelling-based approach using letters.
• Figure 2: (A): The Experiment Design. (B): The trajectory finishing with time. (C): The PSD feature calculated from Raw EEG signal. (D): The structures of the Trajectory Resnet18 Encoder and the Converlutional EEG Encoder. (E): The Letter Probability Distribution in the outputs from the classifier layers. (F): The structure of the sentence generator.
• Figure 3: Model Comparisons: Top 1, Top 3, and Top 5 Accuracy of Different Models. Orange color comparisons shows the difference between w/CL and w/o CL conditions. Asterisks (***) indicate significant results ($\textit{p}~<~0.001$).
• Figure 4: Top 1 Accuracy of CNN w/ CL Model Across Different ROIs and Frequency Bands.
• Figure 5: (A): The KDE distributions among only neural letter classifier (NLC), only LLMs (baseline), and LLMs after NLC. (B): The linear regression between the CER and WER scores of NLC ($S_{NLC}$) and LLMs ($S_{LLMs}$).