Brain Foundation Models with Hypergraph Dynamic Adapter for Brain Disease Analysis

TL;DR

The paper addresses the challenge of generalizing brain analysis models across diverse tasks and modalities. It proposes SAM-Brain3D, a brain-specific 3D foundation model, paired with a Hypergraph Dynamic Adapter HyDA to enable efficient, patient-specific, multi-modal adaptation and multi-scale feature fusion. Empirical results show state-of-the-art performance in brain structure and lesion segmentation, Alzheimer's progression prediction, and MGMT methylation classification, demonstrating strong generalization and transferability. The approach advances brain disease analysis by leveraging multi-modal, high-order relational modeling and dynamic, subject-aware convolution, with potential clinical impact in accurate and adaptable brain diagnostics.

Abstract

Brain diseases, such as Alzheimer's disease and brain tumors, present profound challenges due to their complexity and societal impact. Recent advancements in brain foundation models have shown significant promise in addressing a range of brain-related tasks. However, current brain foundation models are limited by task and data homogeneity, restricted generalization beyond segmentation or classification, and inefficient adaptation to diverse clinical tasks. In this work, we propose SAM-Brain3D, a brain-specific foundation model trained on over 66,000 brain image-label pairs across 14 MRI sub-modalities, and Hypergraph Dynamic Adapter (HyDA), a lightweight adapter for efficient and effective downstream adaptation. SAM-Brain3D captures detailed brain-specific anatomical and modality priors for segmenting diverse brain targets and broader downstream tasks. HyDA leverages hypergraphs to fuse complementary multi-modal data and dynamically generate patient-specific convolutional kernels for multi-scale feature fusion and personalized patient-wise adaptation. Together, our framework excels across a broad spectrum of brain disease segmentation and classification tasks. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art approaches, offering a new paradigm for brain disease analysis through multi-modal, multi-scale, and dynamic foundation modeling.

Figures (4)

• Figure 1: The pipeline of our method. (a) Training the brain foundation model of SAM-Brain3D. SAM-Brain3D is trained on diverse brain datasets and numerous image-label pairs for flexible segmentation targets. It shares the same network architecture as SAM-Med3D wang2023sam which has an image encoder, a prompt encoder, and a decoder. (b) Downstream adaptation with Hypergraph Dynamic Adapter (HyDA) for brain disease diagnosis. This stage uses the parameter-fixed image encoder of SAM-Brain3D to extract feature maps/embeddings from multi-modal imaging data while for non-imaging data (optional), a simple Multi-Layer Perceptron (MLP) can be used as a backbone for feature embedding. The embeddings are exploited to construct modality-specific sub-hypergraphs, concatenated as the final hypergraph. The feature maps, the embeddings (node features), and the final hypergraph are then input to HyDA to obtain predictions and enhanced features, which are fed into a discriminative classifier to predict the disease types. We further average both predictions as the final one.
• Figure 2: The design of Hypergraph Dynamic Adapter (HyDA) for downstream brain disease diagnosis tasks. HyDA exploits two hypergraph convolution layers to extract high-order relations from the final hypergraph and multi-modal feature embeddings (i.e., node features). It then makes a prediction using a hypergraph classifier. Since the semantic prediction of disease types is based on embeddings, they usually contain semantic global context. The semantic embeddings are input to kernel generators, comprising two 1$\times$1$\times$1 convolution layers ('Conv1' and 'Conv2'), to generate semantic and dynamic convolutional kernels for each input subject. The subject-conditioned dynamic kernels then convolve the feature maps of imaging data to fuse the semantic information into the lower-level features, leading to dynamic multi-scale fusion. The fused feature maps are concatenated together and merged using another 1$\times$1$\times$1 convolution ('Conv3'). The merged version will be further enhanced by channel-wise reduction and scale via 'Conv4&5'. The enhanced feature maps are then flattened into a vector and summed with the original node features via a residual connection as the final feature embeddings. The final embeddings of different modalities are then concatenated and input to a discriminative classifier for prediction.
• Figure 3: Ablation on modalities.
• Figure 4: Evaluation on the nearest neighbor $k$.