BrainStack: Neuro-MoE with Functionally Guided Expert Routing for EEG-Based Language Decoding

TL;DR

Decoding linguistic content from EEG is challenging due to low SNR and distributed, nonlinear brain activity. The authors introduce BrainStack, a functionally guided Neuro-Mixture-of-Experts that couples seven anatomically defined regional experts with a global CTNet, coordinated by an adaptive routing gate and enhanced by cross-regional distillation. They also release SilentSpeech-EEG (SS-EEG), a large-scale benchmark with over 120 hours of data from 12 subjects across 24 silent words, enabling robust evaluation. Across within-subject and cross-subject settings, BrainStack achieves state-of-the-art accuracy and generalization, demonstrating that a modular, neuro-inspired MoE approach can effectively leverage distributed cortical signals for brain-language decoding. The work advances interpretable, scalable EEG decoding and provides a valuable resource for future BCI research, including cross-subject robustness and region-aware modeling.

Abstract

Decoding linguistic information from electroencephalography (EEG) remains challenging due to the brain's distributed and nonlinear organization. We present BrainStack, a functionally guided neuro-mixture-of-experts (Neuro-MoE) framework that models the brain's modular functional architecture through anatomically partitioned expert networks. Each functional region is represented by a specialized expert that learns localized neural dynamics, while a transformer-based global expert captures cross-regional dependencies. A learnable routing gate adaptively aggregates these heterogeneous experts, enabling context-dependent expert coordination and selective fusion. To promote coherent representation across the hierarchy, we introduce cross-regional distillation, where the global expert provides top-down regularization to the regional experts. We further release SilentSpeech-EEG (SS-EEG), a large-scale benchmark comprising over 120 hours of EEG recordings from 12 subjects performing 24 silent words, the largest dataset of its kind. Experiments demonstrate that BrainStack consistently outperforms state-of-the-art models, achieving superior accuracy and generalization across subjects. Our results establish BrainStack as a functionally modular, neuro-inspired MoE paradigm that unifies neuroscientific priors with adaptive expert routing, paving the way for scalable and interpretable brain-language decoding.

Figures (5)

• Figure 1: Overall illustration of EEG-based text decoding using BrainStack.
• Figure 2: Overview of the proposed BrainStack for EEG-based text decoding. Raw EEG signals $\mathbf{X} \in \mathbb{R}^{C \times T}$ are partitioned into seven region-specific subsets and one global input. Local experts adopt lightweight spatio-temporal CNNs, while the global expert (CTNet) combines CNN and Transformer layers. Outputs are adaptively fused by an expert routing mechanism $f_{\text{meta}}$ for final prediction $y \in \mathcal{Y}$.
• Figure 3: Human attention dynamics during silent speech execution. Temporal evolution of brain activation maps obtained through attribution analysis. The maps illustrate the spatiotemporal distribution of sensitivity in response to the input stimulus across different brain regions from 100ms to 900ms. Warmer colors represent higher sensitivity, indicating stronger contribution to the model's decision at each time step. Each vertical pair of cortical maps represents two complementary views of the brain at the same moment, offering a clearer visualization of region-specific activations. The analysis captures the progression of localized neural responses transitioning into broader inter-regional integration, revealing both independent regional dynamics and global coordination underlying silent speech decoding.
• Figure 4: Regional contribution analysis in the adaptive routing gate. (a) Correlation between average fusion weight of each region and subject-level accuracy, with fitted regression lines and Pearson’s $r$. (b) Violin plots of weight distributions across experts, overlaid with subject accuracies (color-coded), highlighting variability in regional importance.
• Figure 5: Subject-wise performance variability across three random seeds. Top: classification accuracy with min–max error bands. Bottom: F1-score with corresponding min–max range.