Unlocking Non-Invasive Brain-to-Text

TL;DR

This paper addresses the challenge of non-invasive brain-to-text by proposing a pipeline that extends word-level decoding to sequences using large-language-model rescoring, enables open-vocabulary transcription through predictive in-filling of OOV words, and scales performance via selective pooling of multiple MEG/EEG datasets. The approach achieves the first non-invasive B2T results that exceed chance across standard metrics, with BLEU improvements up to 2.6x, and demonstrates that dataset quality and vocabulary design critically influence performance. Ablation studies confirm the contribution of each component, and selective data pooling yields up to 2.3x gains in word classification accuracy, addressing the longstanding data bottleneck. Together, these advances move non-invasive B2T closer to practical BCI applications and outline pathways for scaling through cross-dataset training and semantic-level decoding.

Abstract

Despite major advances in surgical brain-to-text (B2T), i.e. transcribing speech from invasive brain recordings, non-invasive alternatives have yet to surpass even chance on standard metrics. This remains a barrier to building a non-invasive brain-computer interface (BCI) capable of restoring communication in paralysed individuals without surgery. Here, we present the first non-invasive B2T result that significantly exceeds these critical baselines, raising BLEU by $1.4\mathrm{-}2.6\times$ over prior work. This result is driven by three contributions: (1) we extend recent word-classification models with LLM-based rescoring, transforming single-word predictors into closed-vocabulary B2T systems; (2) we introduce a predictive in-filling approach to handle out-of-vocabulary (OOV) words, substantially expanding the effective vocabulary; and (3) we demonstrate, for the first time, how to scale non-invasive B2T models across datasets, unlocking deep learning at scale and improving accuracy by $2.1\mathrm{-}2.3\times$. Through these contributions, we offer new insights into the roles of data quality and vocabulary size. Together, our results remove a major obstacle to realising practical non-invasive B2T systems.

Figures (10)

• Figure 1: Our approach outperforms all non-invasive B2T methods. Ranks are calculated from absolute improvement over associated random baselines.
• Figure 2: Brain-to-text decoding method. We pool data from multiple heterogeneous datasets (e.g. A and B) and align brain data segments to word onsets. Then, the segments are encoded by a signal encoder, handling dataset differences. A transformer learns the relationships between the encoded latents, embedding them with context. Its outputs are predictions of target word embeddings from a large language model. We map these predictions into logit distributions over the target vocabulary and a beam search with contextual rescoring constructs the highest probability sentence. If we detect out-of-vocabulary words in the stimulus, an in-filling model inserts words into these positions.
• Figure 3: Selectively pooling data improves accuracy by $\mathbf{2.3\times}$.(A) Pairing datasets in training. Numbers show accuracy improvement on the evaluation dataset when trained additionally with the training dataset. Numbers in brackets show raw accuracy (asterisks are statistically significant against chance). The diagonal shows no paired training, i.e. standalone. Shading shows the change in accuracy relative to standalone. (B) Exploiting selective pooling doubles accuracy. We combine the target data (Gwilliams and Broderick) with the best, and the best and second best, datasets by quality, i.e. LibriBrain and Armeni, leading to dramatic improvements on the target data.
• Figure 4: Optimal vocabulary size differs by method and metric. We train our model with increasingly large vocabularies and show the effect on each decoding strategy's performance. Methods with/without in-filling tend to converge with larger vocabularies as there are fewer words to in-fill.
• Figure 5: Alignment is critical. We add a random jitter in the range $[0, \mathrm{jitter}]$ to the aligned input samples. We train on LibriBrain with a vocabulary size of 250 and quote word classification accuracy.