Brain-MGF: Multimodal Graph Fusion Network for EEG-fMRI Brain Connectivity Analysis Under Psilocybin

TL;DR

Brain-MGF tackles the problem of decoding psilocybin-induced brain states by integrating EEG and fMRI connectivity through adaptive multimodal graphs. It uses modality-specific GraphConv encoders on top of 40% sparse partial-correlation graphs with Pearson-based node features plus a learnable pseudo-identity projection, fused by a sample-specific gate: $\mathbf{z}_{\mathrm{fus}} = \alpha_{\mathrm{fMRI}} \mathbf{z}_{\mathrm{fMRI}} + \alpha_{\mathrm{EEG}} \mathbf{z}_{\mathrm{EEG}}$. On the PsiConnect dataset, Brain-MGF outperformed unimodal and non-adaptive fusion across meditation and rest (e.g., $74.02\%$ accuracy and $75.72\%$ ROC-AUC for meditation; $76.00\%$ accuracy and $85.83\%$ ROC-AUC for rest), with UMAP visualisations showing more separable fused embeddings. The gating analysis indicates that fMRI provides a stable backbone while EEG contributes contextual information, offering an interpretable framework for cross-modal neural integration under psychedelics.

Abstract

Psychedelics, such as psilocybin, reorganise large-scale brain connectivity, yet how these changes are reflected across electrophysiological (electroencephalogram, EEG) and haemodynamic (functional magnetic resonance imaging, fMRI) networks remains unclear. We present Brain-MGF, a multimodal graph fusion network for joint EEG-fMRI connectivity analysis. For each modality, we construct graphs with partial-correlation edges and Pearson-profile node features, and learn subject-level embeddings via graph convolution. An adaptive softmax gate then fuses modalities with sample-specific weights to capture context-dependent contributions. Using the world's largest single-site psilocybin dataset, PsiConnect, Brain-MGF distinguishes psilocybin from no-psilocybin conditions in meditation and rest. Fusion improves over unimodal and non-adaptive variants, achieving 74.0% accuracy and 76.5% F1 score on meditation, and 76.0% accuracy with 85.8% ROC-AUC on rest. UMAP visualisations reveal clearer class separation for fused embeddings. These results indicate that adaptive graph fusion effectively integrates complementary EEG-fMRI information, providing an interpretable framework for characterising psilocybin-induced alterations in large-scale neural organisation.

Figures (2)

• Figure 1: Brain--MGF architecture. Each modality (fMRI and EEG) constructs a connectivity graph from Pearson and partial correlations, combined with a pseudo-identity projection to form node features $\mathbf{X}$ and edges $\mathbf{A}$. Three GraphConv blocks with LeakyReLU and dropout encode node representations, followed by global mean pooling to obtain subject-level embeddings $\mathbf{z}_{\mathrm{fMRI}}$ and $\mathbf{z}_{\mathrm{EEG}}$. An adaptive gating MLP computes softmax weights $\alpha_{\mathrm{fMRI}}, \alpha_{\mathrm{EEG}}\!\in\!(0,1)$ to fuse modalities: $\mathbf{z}_{\mathrm{fus}}=\alpha_{\mathrm{fMRI}}\mathbf{z}_{\mathrm{fMRI}}+\alpha_{\mathrm{EEG}}\mathbf{z}_{\mathrm{EEG}}$. The fused embedding is normalised with LayerNorm and passed through an MLP classifier with sigmoid output, optimised using binary cross-entropy loss.
• Figure 2: UMAP visualisation of learned embeddings during meditation. Each point represents a subject; colour denotes condition (no-psilocybin vs. psilocybin). EEG embeddings show overlapping clusters, fMRI displays clearer separation, and fusion achieves the most distinct manifolds, reflecting its quantitative advantage in Table \ref{['tab:benchmark_meditation']} and Table \ref{['tab:results_psiconnect']}.