MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Training

TL;DR

MEG-XL tackles data inefficiency in brain-to-text by pre-training a long-context MEG model on 2.5-minute windows to learn extended neural priors. It uses a BioCodec-based tokenizer and a criss-cross transformer with sensor-aware embeddings, trained with masked token prediction over long MEG sequences. Fine-tuning on contextual word decoding across three datasets shows substantial gains in data-limited regimes and competitive performance with abundant per-subject data, outperforming several brain foundation models. The work reveals that longer pre-training context yields richer representations and hierarchical attention, suggesting long-context priors enable better cross-subject transfer and zero-shot predictions, with important implications for non-invasive clinical BCIs.

Abstract

Clinical brain-to-text interfaces are designed for paralysed patients who cannot provide extensive training recordings. Pre-training improves data-efficient generalisation by learning statistical priors across subjects, but these priors critically depend on context. While natural speech might unfold gradually over minutes, most methods pre-train with only a few seconds of context. Thus, we propose MEG-XL, a model pre-trained with 2.5 minutes of MEG context per sample, 5-300x longer than prior work, and equivalent to 191k tokens, capturing extended neural context. Fine-tuning on the task of word decoding from brain data, MEG-XL matches supervised performance with a fraction of the data (e.g. 1hr vs 50hrs) and outperforms brain foundation models. We find that models pre-trained with longer contexts learn representations that transfer better to word decoding. Our results indicate that long-context pre-training helps exploit extended neural context that other methods unnecessarily discard. Code, model weights, and instructions are available at https://github.com/neural-processing-lab/MEG-XL .

Figures (8)

• Figure 1: MEG-XL introduces long-context MEG pre-training. When fine-tuned, this approach generalises to decoding words in brain-to-text with less labelled subject data than required by the supervised state-of-the-art (SOTA) and brain foundation models (FMs).
• Figure 2: Overview of the MEG-XL pre-training framework. (Left) A frozen BioCodec tokenizer independently encodes each MEG channel into discrete tokens across Q=6 residual quantization levels, providing prediction targets for self-supervised learning. (Middle) Token embeddings, which are concatenated across quantization levels and projected, are combined with sensor position, orientation, and type embeddings, then processed by a criss-cross transformer. A projection head maps transformer embeddings back to Q residual tokens. (Right) In pre-training, we randomly mask contiguous 3-second blocks uniformly across all sensors, forcing the model to predict masked tokens from temporal context rather than interpolating across channels, until 40% of tokens are masked.
• Figure 3: Pre-training enables generalisation with less subject data. We compare MEG-XL to the state-of-the-art supervised method dAscoli2025TowardsDI and a baseline trained from scratch (MEG-XL with random init.) across varying amounts of fine-tuning data. MEG-XL consistently outperforms its randomly initialised counterpart, confirming that the gains stem from learned priors rather than architecture alone. On Armeni and MEG-MASC, where per-subject data is shallower, MEG-XL outperforms dAscoli2025TowardsDI throughout most of the data range. On LibriBrain, a deep single-subject dataset, both methods perform similarly until approximately 2.5 hours of training data, after which their supervised method pulls ahead. This suggests pre-training may substitute for subject-specific data as when recordings are scarce, learned priors help. In rare cases where recordings are abundant, learning from scratch eventually wins.
• Figure 4: Linear probing shows that models pre-trained with more context generalise better to word decoding. We pre-train models with increasing context, fixed masking percentage, and constant optimisation steps, then evaluate the strength of their representations with linear probes (frozen backbone). We compare two conditions: full context, where all models see 150s of input to isolate representation quality, and matched context, where the input is restricted to the pre-training length. The lack of divergence between the lines suggests models cannot leverage inference context that exceeds their pre-training context. We train the linear probes with 7% of the training data. We could not expand further than 150s due to GPU memory limits. Token-matched pre-training shows similar trends (Appendix \ref{['app:tokenmatch']}).
• Figure 5: (Top) Extending neural context improves zero-shot prediction of brain activity from unseen datasets and subjects. We mask the central 3s subsegment of samples from unseen datasets and measure improvement in token prediction accuracy (relative to chance) of models pre-trained on increasing neural context. Scaling improves masked prediction, with the trend remaining through 150s. Only GPU VRAM limits prevent increasing it further. Chance accuracy is $1/256$. (Bottom) Long-context pretraining induces selective and hierarchical attention. (Left) Models pretrained on longer context attend locally in early layers before expanding to integrate distant context; short-context models attend diffusely throughout. (Right) Attention entropy decreases with context length, indicating more selective attention patterns. We provide 150s context at inference. See Appendix \ref{['app:attentionanalysis']} for attention distance and entropy calculations.