Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

TL;DR

This work proposes Brain-OF, the first omnifunctional brain foundation model jointly pretrained on fMRI, EEG and MEG, capable of handling both unimodal and multimodal inputs within a unified framework, and introduces the Any-Resolution Neural Signal Sampler, which projects diverse brain signals into a shared semantic space.

Abstract

Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited to a single functional modality, restricting their ability to exploit complementary spatiotemporal dynamics and the collective data scale across imaging techniques. To address this limitation, we propose Brain-OF, the first omnifunctional brain foundation model jointly pretrained on fMRI, EEG and MEG, capable of handling both unimodal and multimodal inputs within a unified framework. To reconcile heterogeneous spatiotemporal resolutions, we introduce the Any-Resolution Neural Signal Sampler, which projects diverse brain signals into a shared semantic space. To further manage semantic shifts, the Brain-OF backbone integrates DINT attention with a Sparse Mixture of Experts, where shared experts capture modality-invariant representations and routed experts specialize in modality-specific semantics. Furthermore, we propose Masked Temporal-Frequency Modeling, a dual-domain pretraining objective that jointly reconstructs brain signals in both the time and frequency domains. Brain-OF is pretrained on a large-scale corpus comprising around 40 datasets and demonstrates superior performance across diverse downstream tasks, highlighting the benefits of joint multimodal integration and dual-domain pretraining.

Figures (7)

• Figure 1: Overview of the Brain-OF Architecture. Brain-OF is an omnifunctional foundation model jointly pretrained on fMRI, EEG and MEG. (a) Pretraining pipeline: The original signals are masked in both the temporal and frequency domains to encourage the model to jointly learn coupled time–frequency representations. (b) ARNESS projects arbitrary resolution signals into a unified semantic space and also serves as a decoder to inverse-sample latent representations for reconstruction during pretraining. (c) Multimodal Fusion serially resamples multiple unimodal sequences via ARNESS without introducing modality-specific fusion branches. (d) Sparse MoE addresses semantic heterogeneity by dynamically routing tokens to specialized experts while extracting modality-invariant knowledge through shared experts. (e) DINT Attention mitigates attention allocation to irrelevant contextual information. Here, sFFT and tFFT denote Fast Fourier Transform along spatial and temporal dimensions; iFFT denotes the inverse Fast Fourier Transform; and M indicates the masking.
• Figure 2: Visualization of Brain-OF Interpretability. (a) EEG channel contribution topomap for abnormality detection on TUAB. (b) top 10% most influential AD-related brain regions on ADNI. (c) MEG sensor contribution map for brain-age prediction on CamCAN.
• Figure 3: Modality Importance Analysis. (a) Local Modality Importance. Heatmap showing the relative performance drop (%) on each downstream dataset when a specific modality (fMRI, EEG, or MEG) is removed during pretraining. Higher values indicate stronger reliance on that modality. (b) Global Modality Importance. Importance scores aggregated across all downstream tasks, illustrating that EEG and fMRI contribute most to the overall generalization capability of Brain-OF, while MEG provides complementary support.
• Figure 4: The impact of the router bias update rate $\gamma$ during pretraining. The performances are evaluated on downstream tasks across three modalities.
• Figure 5: Qualitative visualization of signal reconstruction across fMRI, EEG and MEG.