Integrated Brain Connectivity Analysis with fMRI, DTI, and sMRI Powered by Interpretable Graph Neural Networks

TL;DR

The paper presents an integrated multimodal neuroimaging framework that fuses FC, SC, and anatomical features using a MaskGNN with learnable edge masks to achieve interpretable brain connectivity analysis. By leveraging the Glasser atlas and HCP-D developmental data, the method improves cognitive score predictions and enables identification of key brain networks via Grad-RAM/Grad-CAM analyses. The approach demonstrates benefits of multimodal fusion (FC, SC, AS) and manifold/mask regularization, providing both accurate predictions and interpretable insights into structure–function coupling during development. The work highlights the potential of edge-level interpretability in graph-based neuroimaging models and discusses limitations and avenues for future enhancements, including advanced fusion techniques and broader populations.

Abstract

Multimodal neuroimaging modeling has becomes a widely used approach but confronts considerable challenges due to heterogeneity, which encompasses variability in data types, scales, and formats across modalities. This variability necessitates the deployment of advanced computational methods to integrate and interpret these diverse datasets within a cohesive analytical framework. In our research, we amalgamate functional magnetic resonance imaging, diffusion tensor imaging, and structural MRI into a cohesive framework. This integration capitalizes on the unique strengths of each modality and their inherent interconnections, aiming for a comprehensive understanding of the brain's connectivity and anatomical characteristics. Utilizing the Glasser atlas for parcellation, we integrate imaging derived features from various modalities: functional connectivity from fMRI, structural connectivity from DTI, and anatomical features from sMRI within consistent regions. Our approach incorporates a masking strategy to differentially weight neural connections, thereby facilitating a holistic amalgamation of multimodal imaging data. This technique enhances interpretability at connectivity level, transcending traditional analyses centered on singular regional attributes. The model is applied to the Human Connectome Project's Development study to elucidate the associations between multimodal imaging and cognitive functions throughout youth. The analysis demonstrates improved predictive accuracy and uncovers crucial anatomical features and essential neural connections, deepening our understanding of brain structure and function.

Figures (9)

• Figure 1: The depiction of the proposed framework: Functional connectivity (FC) and structural connectivity (SC) obtained from fMRI and DTI, respectively, are amalgamated at the nodal level and subsequently fed into the MaskGNN for predictive analysis. In the latent space, embeddings of nodal features are integrated with anatomical statistics (AS) from sMRI, alongside a computation of structure-functional coupling using the FC and SC matrices. The aggregated features are then subjected to MaskGNN embedding, graph pooling, and readout processes. After post-training, the visualization of the uniform mask across MaskGNN layers is achieved, and a post-hoc approach is used to elucidate the contribution of AS.
• Figure 2: The age distribution of selected subjects.
• Figure 3: The distribution of intelligence metrics.
• Figure 4: The use of Grad-CAM and Grad-RAM scores for model explainability: (a) Grad-RAM scores for simultaneous prediction of CCC and FCC; (b) Discrimination of groups using Grad-CAM scores across distinct TCC levels
• Figure 5: A comparative analysis of predictive performance showing the individual and combined effects of the manifold regularization term ($L_{manifold}$) and mask penalty ($L_{mask}$) on the proposed model, with a baseline scenario for reference. All comparisons are supported by pair-wise t-tests against MGNN, with p-values displayed above each bar except for MGNN, emphasizing significant differences.