Assembling the Mind's Mosaic: Towards EEG Semantic Intent Decoding

TL;DR

This work tackles the challenge of natural, interpretable brain-to-language communication by introducing Semantic Intent Decoding (SID), which represents communicative meaning as a compositional set of semantic units. The BrainMosaic architecture implements SID through a three-stage pipeline: a Semantic Decomposer that parses neural signals into a variable-sized semantic unit set, a Semantic Retriever that aligns these units to an open continuous semantic space, and a Semantic Decoder that reconstructs fluent sentences via a constrained, semantic-grounded generation process. The approach improves generalization, scalability, and interpretability over fixed-label and end-to-end paradigms, as demonstrated on multilingual EEG and clinical SEEG datasets with robust ablations and open-vocabulary experiments. These results advance open-vocabulary, semantically grounded BCIs and highlight the value of interpretable intermediate representations for translating neural activity into natural language messages with high fidelity.

Abstract

Enabling natural communication through brain-computer interfaces (BCIs) remains one of the most profound challenges in neuroscience and neurotechnology. While existing frameworks offer partial solutions, they are constrained by oversimplified semantic representations and a lack of interpretability. To overcome these limitations, we introduce Semantic Intent Decoding (SID), a novel framework that translates neural activity into natural language by modeling meaning as a flexible set of compositional semantic units. SID is built on three core principles: semantic compositionality, continuity and expandability of semantic space, and fidelity in reconstruction. We present BrainMosaic, a deep learning architecture implementing SID. BrainMosaic decodes multiple semantic units from EEG/SEEG signals using set matching and then reconstructs coherent sentences through semantic-guided reconstruction. This approach moves beyond traditional pipelines that rely on fixed-class classification or unconstrained generation, enabling a more interpretable and expressive communication paradigm. Extensive experiments on multilingual EEG and clinical SEEG datasets demonstrate that SID and BrainMosaic offer substantial advantages over existing frameworks, paving the way for natural and effective BCI-mediated communication.

Figures (3)

• Figure 1: Overview of the Semantic Intent Decoding (SID) framework.
• Figure 2: Qualitative examples of two-level evaluation metrics.Underlines mark meaningful Chinese or English word groups. (Parentheses indicate additional elements inserted in English translations for syntactic or semantic clarity.) (a) Concept (word) level examples are sampled from the union of the Clinical + Chisco + the 30k Chinese dictionary. Each word falls into one of the embedding similarity intervals [0, 0.5), [0.5, 0.6), [0.6, 0.7), [0.7, 0.8), [0.8, 1.0) relative to the target units in the Doubao embedding space. (b) Intent (sentence) level examples including the gold reference, one prediction from BrainMosaic, five baseline results and two manually constructed sentences, illustrating typical linguistic phenomena (sorted by SRS score in descending order).
• Figure 3: Regional contribution analysis. (a,b) Spatial distribution of electrodes and SEEG channels on the MNI template brain (top and left views). (c) Gradient-based channel saliency, with red indicating higher contribution.. (d) Positive correlations ($P \le 0.01$) between electrode saliency and single-electrode decoding performance (UMA/MUS).