Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes

TL;DR

Brain-to-speech BCIs decode speech from neural activity across vocalized, mimed, and imagined modes, yet the internal representations enabling cross-mode decoding are not well understood. The authors apply mechanistic interpretability methods—activation patching, tri-modal interpolation, causal tracing, and scrubbing—on a stereotactic EEG decoder trained on three speech modes to causally probe internal representations. They find that speech modes lie on a shared, continuous causal manifold and that cross-mode transfer is mediated by compact, layer-specific subspaces, with early-to-mid recurrent dynamics and a small convolutional subspace driving the effects. These results provide a principled basis for cross-mode generalization and interpretable, robust brain-to-speech BCIs by emphasizing structured subspaces and directional transfer patterns.

Abstract

Brain-to-speech decoding models demonstrate robust performance in vocalized, mimed, and imagined speech; yet, the fundamental mechanisms via which these models capture and transmit information across different speech modalities are less explored. In this work, we use mechanistic interpretability to causally investigate the internal representations of a neural speech decoder. We perform cross-mode activation patching of internal activations across speech modes, and use tri-modal interpolation to examine whether speech representations vary discretely or continuously. We use coarse-to-fine causal tracing and causal scrubbing to find localized causal structure, allowing us to find internal subspaces that are sufficient for cross-mode transfer. In order to determine how finely distributed these effects are within layers, we perform neuron-level activation patching. We discover that small but not distributed subsets of neurons, rather than isolated units, affect the cross-mode transfer. Our results show that speech modes lie on a shared continuous causal manifold, and cross-mode transfer is mediated by compact, layer-specific subspaces rather than diffuse activity. Together, our findings give a causal explanation for how speech modality information is organized and used in brain-to-speech decoding models, revealing hierarchical and direction-dependent representational structure across speech modes.

Figures (8)

• Figure 1: Multi-channel sEEG signals are processed by a convolutional encoder to extract spatiotemporal features, followed by a bidirectional recurrent network that integrates information over time and predicts frame-level mel-spectrograms. A neural vocoder converts predicted spectrograms into reconstructed speech.
• Figure 2: Mechanistic interpretability across speech modes.(1) We study speech decoding under three modes: vocalized, mimed, and imagined, recorded with sEEG. (2)Cross-mode activation patching: we replace internal activations from one mode with those from another at selected layers (e.g., CNN/RNN) to test whether representations transfer across modes and whether patching yields stronger or cleaner imagination signals. (3)Tri-modal activation interpolation: we linearly interpolate activations between modes to probe smooth transitions in representational geometry and decoding behavior. (4)Temporal localization: we causally localize when information matters by patching subsequent convolution channels and contiguous early/middle/late thirds of the RNN activation sequence (coarse), and by sliding a 25% window with 25% shift (fine-grained), measuring the resulting change in decoding performance. (5)Neuron-level cross-modal transfer: we test whether transfer is mediated by single neurons, restricted subsets, or broad populations by patching individual neurons and then patching top-$k$ neurons grouped by their effects.
• Figure 3: RNN sliding-window causal tracing across speech modes. Early-to-mid recurrent states form a causal bottleneck that is both sufficient and necessary for high-quality speech decoding.
• Figure 4: Causal subspace concentration in the convolutional encoder. Ranked Top-$k$ channel subgroup patching consistently outperforms matched random-$k$ controls, demonstrating that cross-mode transfer is mediated by a compact convolutional subspace rather than uniformly distributed features.
• Figure 5: Top-k neuron patching reveals asymmetric subspace transfer across speech modalities. $\Delta$PCC (relative to k = 1) is shown as a function of the number of patched neurons for three target modalities: Vocalized (left), Mimed (middle), and Imagined (right). Solid lines denote RNN layers and dashed lines denote convolutional layers. Shaded regions indicate $\pm$ SEM across folds.