Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Connectome Fusion via Cross-Attention for Autism Spectrum Disorder Classification Using Graph Learning

Ansar Rahman, Hassan Shojaee-Mend, Sepideh Hatamikia

Abstract

Autism spectrum disorder (ASD) is a complex neurodevelopmental condition characterized by atypical functional brain connectivity and subtle structural alterations. rs-fMRI has been widely used to identify disruptions in large-scale brain networks, while structural MRI provides complementary information about morphological organization. Despite their complementary nature, effectively integrating these heterogeneous imaging modalities within a unified framework remains challenging. This study proposes a multimodal graph learning framework that preserves the dominant role of functional connectivity while integrating structural imaging and phenotypic information for ASD classification. The proposed framework is evaluated on ABIDE-I dataset. Each subject is represented as a node within a population graph. Functional and structural features are extracted as modality-specific node attributes, while inter-subject relationships are modeled using a pairwise association encoder (PAE) based on phenotypic information. Two Edge Variational GCNs are trained to learn subject-level embeddings. To enable effective multimodal integration, we introduce a novel asymmetric transformer-based cross-attention mechanism that allows functional embeddings to selectively incorporate complementary structural information while preserving functional dominance. The fused embeddings are then passed to a MLP for ASD classification. Using stratified 10-fold cross-validation, the framework achieved an AUC of 87.3% and an accuracy of 84.4%. Under leave-one-site-out cross-validation (LOSO-CV), the model achieved an average cross-site accuracy of 82.0%, outperforming existing methods by approximately 3% under 10-fold cross-validation and 7% under LOSO-CV. The proposed framework effectively integrates heterogeneous multimodal data from the multi-site ABIDE-I dataset, improving automated ASD classification across imaging sites.

Multimodal Connectome Fusion via Cross-Attention for Autism Spectrum Disorder Classification Using Graph Learning

Abstract

Autism spectrum disorder (ASD) is a complex neurodevelopmental condition characterized by atypical functional brain connectivity and subtle structural alterations. rs-fMRI has been widely used to identify disruptions in large-scale brain networks, while structural MRI provides complementary information about morphological organization. Despite their complementary nature, effectively integrating these heterogeneous imaging modalities within a unified framework remains challenging. This study proposes a multimodal graph learning framework that preserves the dominant role of functional connectivity while integrating structural imaging and phenotypic information for ASD classification. The proposed framework is evaluated on ABIDE-I dataset. Each subject is represented as a node within a population graph. Functional and structural features are extracted as modality-specific node attributes, while inter-subject relationships are modeled using a pairwise association encoder (PAE) based on phenotypic information. Two Edge Variational GCNs are trained to learn subject-level embeddings. To enable effective multimodal integration, we introduce a novel asymmetric transformer-based cross-attention mechanism that allows functional embeddings to selectively incorporate complementary structural information while preserving functional dominance. The fused embeddings are then passed to a MLP for ASD classification. Using stratified 10-fold cross-validation, the framework achieved an AUC of 87.3% and an accuracy of 84.4%. Under leave-one-site-out cross-validation (LOSO-CV), the model achieved an average cross-site accuracy of 82.0%, outperforming existing methods by approximately 3% under 10-fold cross-validation and 7% under LOSO-CV. The proposed framework effectively integrates heterogeneous multimodal data from the multi-site ABIDE-I dataset, improving automated ASD classification across imaging sites.
Paper Structure (21 sections, 11 equations, 5 figures, 5 tables)

This paper contains 21 sections, 11 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Proposed Methodology
  3. Dataset and Preprocessing
  4. Multimodal Brain Network Construction
  5. Feature Selection and Standardization
  6. Adaptive Population Graph Construction
  7. Pairwise Association Encoder (PAE)
  8. Edge-Variational Graph Convolution on Adaptive Population Graphs
  9. EV-GCN Encoder
  10. Transformer-based Asymmetric Cross-Attention for Multimodal Fusion
  11. Asymmetric Cross-Attention Formulation
  12. Residual Refinement
  13. Classification Module
  14. Experiments and Results
  15. Performance Evaluation
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Overview of multimodal ASD diagnosis using heterogeneous ABIDE multi-site data, highlighting the transition from clinician-based assessment to AI-driven multimodal analysis
  • Figure 2: End-to-End Workflow Diagram(a) rs-fMRI, sMRI, and phenotypic data are processed to extract functional and structural connectivity features; (b) modality-specific population graphs are constructed and processed using EV-GCN to learn functional and structural node embeddings;(c) the resulting embeddings are integrated through asymmetric cross-attention fusion and passed to a Feed Forward MLP classifier to predict ASD vs. TD
  • Figure 3: Transformer-based asymmetric cross-attention fusion integrating modality-specific EV-GCN embeddings from functional and structural brain connectivity.
  • Figure 4: ROC–AUC curve for the proposed model under 10-fold cross-validation
  • Figure 5: Bar plot comparing the performance of the proposed model with SOTA methods under LOSO cross-validation