BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Abstract

Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.

Figures (1)

• Figure 1: A. Our decoding pipelines: the low-compute end-to-end branch uses an encoder–decoder architecture with beam search, while the high-compute branch feeds phoneme logits into a weighted finite-state transducer to generate candidates, which are then rescored. B. BrainWhisperer's architecture. Neural features are first mapped into a shared embedding space using a convolutional front-end (embedder) and then processed by a stack of transformer encoder layers. A phoneme head predicts phoneme sequences using a CTC loss, while the remaining encoder layers feed a transformer decoder trained with cross-entropy on text tokens. C. The proposed month-specific and day-specific low-rank projections. D. Our windowed attention mechanism. Windowing is applied to the phoneme-selective encoder layers.