Decoding Linguistic Representations of Human Brain

TL;DR

The paper surveys the decoding of linguistic representations from human brain activity, framing the problem as a cross-disciplinary endeavor that combines neuroscience with deep learning. It provides a taxonomy of brain-to-language decoding tasks, from brain-network alignment and neural encoding to textual and speech Stimuli Recognition, brain recording translation, and speech neuroprosthesis. It discusses evaluation metrics, datasets, and architectural patterns, highlighting the progression from text classification to inner speech and open-vocabulary brain-to-speech approaches, including invasive and non-invasive data with LLMs playing an increasingly central role. The work emphasizes practical implications for BCIs, particularly for ALS patients, and outlines future directions such as universal decoders, multi-modality integration, and ethical considerations, aiming to accelerate research at the intersection of neuroscience and AI. The analysis underlines the potential to extend brain decoding toward more naturalistic and high-bandwidth communication, bridging neural activity and sophisticated linguistic outputs.

Abstract

Language, as an information medium created by advanced organisms, has always been a concern of neuroscience regarding how it is represented in the brain. Decoding linguistic representations in the evoked brain has shown groundbreaking achievements, thanks to the rapid improvement of neuroimaging, medical technology, life sciences and artificial intelligence. In this work, we present a taxonomy of brain-to-language decoding of both textual and speech formats. This work integrates two types of research: neuroscience focusing on language understanding and deep learning-based brain decoding. Generating discernible language information from brain activity could not only help those with limited articulation, especially amyotrophic lateral sclerosis (ALS) patients but also open up a new way for the next generation's brain-computer interface (BCI). This article will help brain scientists and deep-learning researchers to gain a bird's eye view of fine-grained language perception, and thus facilitate their further investigation and research of neural process and language decoding.

Figures (7)

• Figure 1: The main content flow and categorization of this survey.
• Figure 2: The formation of linguistic representation in the human brain. a. The human brain tracks the dynamic flow of speech and linguistic properties with minor response delay, and the neural response is performed in a continuous predictive manner. b. The human brain and the neural networks can both encode textual or verbal-linguistic stimuli into specific representations, and the decoding process aims to generate its original form from the evoked response. c. The scaling laws for the brain encoding models and pre-trained LLMs respectively antonello2024scalinglin2024selecting. For brain encoding, $S_1$, $S_2$ and $S_3$ represent different experiment subjects. The performance grows as the model parameters increase.
• Figure 3: Stimuli recognition of evoked brain activity. a. An overview of the stimuli recognition task. The subject receives textual or vocal information while the active brain signals are collected. The raw brain recordings are processed into feature space, followed by classifiers, networks or pre-trained models to distinguish the original stimuli based on the complexity and candidate size. Several approaches adopted Word embeddings (i.e. word2vec) to compare the decoded vector in a semantic space. b. In natural listening scenarios, restoring the original speech features and waveform is a more complex task. Regression models (i.e. ridge regression), CNN and RNN-based network modules, and paramount generation models (i.e. GAN) are widely used. c. The decoding architecture for various speech-related targets. The speech envelope can be easily reconstructed with CNNs while more complex networks are necessary for the decoding of MFCC accou2023decodingpetrosyan2021compact. The most difficult task is to synthesize the stimuli wave, where an encoder-generator-vocoder architecture has been verified effective wang2020stimulus.
• Figure 4: The experiment setting and model architecture of brain recording translation. a. Taking the natural reading scenario as an example, the subject performs natural reading while the active brain signals are collected. The eye movements are typically recorded to determine the text transcription corresponding to the brain data at each time step. A sequence-to-sequence model processes the evoked brain recordings to determine the related word and then form the decoded sentence. b. A feasible translation model architecture, including feature extraction, feature transformation and a pre-trained encoder-decoder to generate the decoding sentence. Both the pre-trained language models (i.e. BART) and speech models (i.e. Whisper) have been verified effective.
• Figure 5: Overview of speech neuroprosthesis. a. The experiment setting for inner speech recognition. The subjects attempt to speak without making a sound (inner speech) or try their best to pronounce (overt speech), while their active brain signals are collected. From the perspective of neurological, the brain controls the movement of the articulatory system to complete the pronunciation of each phoneme in series, thereby producing recognizable speech, indicating the mapping from evoked brain signals to movements of the articulators to phonemes. The classification and the recognition module are adopted to generate the corresponding phoneme sequences before leveraging the language model to form word sequences.b. The comparison between ASR and inner speech recognition (ISR). The raw time-series signals are processed for feature extraction and then fed into acoustic and brain models respectively. Both models aim to bridge the relationship between learnable features related to acoustics and phoneme sequences. The Viterbi decoding algorithm is performed on the sum of the phoneme probability from the acoustic/brain model and the language probability derived from a language model trained on an extensive corpus to generate the decoded word sequences. c. The brain model can be implemented to decode various modalities. For inner speech recognition, the phoneme and word sequences are decoded with the aim of language models. For brain-to-speech decoding, the speech waves are synthesized according to the articulator gestures, synthesizer parameters or speech properties. By modeling the articulator gesture probability and adopting a gesture-animation system, the talking head can be generated. Different modalities are associated through TTS, ASR, talking head generation (THG) and synthesis methods. d. The acoustic-related brain activities show the potential to develop communication-aided BCI for ALS patients considering the decoding feasibility of text, speech and facial expression.