Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning 3D Medical Image Models From Brain Functional Connectivity Network Supervision For Mental Disorder Diagnosis

Xingcan Hu, Wei Wang, Li Xiao

TL;DR

This paper introduces CINP, a contrastive pre-training framework that aligns 3D T1-weighted MRI with functional connectivity networks derived from fMRI to improve mental disorder diagnosis. It combines image-network contrastive learning, masked image modeling, and image-network matching to learn unified representations, and adds a network prompting strategy to diagnose using mostly sMRI with limited FCN data. Across ADHD, ASD, and multi-class SRPBS tasks, CINP demonstrates competitive accuracy and showcases the potential of incorporating readily available structural MRI into clinical workflows. The approach highlights how multimodal cognitive-neural information can be fused at pre-training to boost diagnostic performance, with practical benefits in low-data clinical settings.

Abstract

In MRI-based mental disorder diagnosis, most previous studies focus on functional connectivity network (FCN) derived from functional MRI (fMRI). However, the small size of annotated fMRI datasets restricts its wide application. Meanwhile, structural MRIs (sMRIs), such as 3D T1-weighted (T1w) MRI, which are commonly used and readily accessible in clinical settings, are often overlooked. To integrate the complementary information from both function and structure for improved diagnostic accuracy, we propose CINP (Contrastive Image-Network Pre-training), a framework that employs contrastive learning between sMRI and FCN. During pre-training, we incorporate masked image modeling and network-image matching to enhance visual representation learning and modality alignment. Since the CINP facilitates knowledge transfer from FCN to sMRI, we introduce network prompting. It utilizes only sMRI from suspected patients and a small amount of FCNs from different patient classes for diagnosing mental disorders, which is practical in real-world clinical scenario. The competitive performance on three mental disorder diagnosis tasks demonstrate the effectiveness of the CINP in integrating multimodal MRI information, as well as the potential of incorporating sMRI into clinical diagnosis using network prompting.

Learning 3D Medical Image Models From Brain Functional Connectivity Network Supervision For Mental Disorder Diagnosis

TL;DR

This paper introduces CINP, a contrastive pre-training framework that aligns 3D T1-weighted MRI with functional connectivity networks derived from fMRI to improve mental disorder diagnosis. It combines image-network contrastive learning, masked image modeling, and image-network matching to learn unified representations, and adds a network prompting strategy to diagnose using mostly sMRI with limited FCN data. Across ADHD, ASD, and multi-class SRPBS tasks, CINP demonstrates competitive accuracy and showcases the potential of incorporating readily available structural MRI into clinical workflows. The approach highlights how multimodal cognitive-neural information can be fused at pre-training to boost diagnostic performance, with practical benefits in low-data clinical settings.

Abstract

In MRI-based mental disorder diagnosis, most previous studies focus on functional connectivity network (FCN) derived from functional MRI (fMRI). However, the small size of annotated fMRI datasets restricts its wide application. Meanwhile, structural MRIs (sMRIs), such as 3D T1-weighted (T1w) MRI, which are commonly used and readily accessible in clinical settings, are often overlooked. To integrate the complementary information from both function and structure for improved diagnostic accuracy, we propose CINP (Contrastive Image-Network Pre-training), a framework that employs contrastive learning between sMRI and FCN. During pre-training, we incorporate masked image modeling and network-image matching to enhance visual representation learning and modality alignment. Since the CINP facilitates knowledge transfer from FCN to sMRI, we introduce network prompting. It utilizes only sMRI from suspected patients and a small amount of FCNs from different patient classes for diagnosing mental disorders, which is practical in real-world clinical scenario. The competitive performance on three mental disorder diagnosis tasks demonstrate the effectiveness of the CINP in integrating multimodal MRI information, as well as the potential of incorporating sMRI into clinical diagnosis using network prompting.

Paper Structure

This paper contains 19 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Contrastive Image-Network Pre-training
  4. Image-Network Contrastive Learning.
  5. Masked Image Modeling.
  6. Image-Network Matching.
  7. Network prompting
  8. Experiments
  9. Experimental Settings
  10. Datasets and Preprocessing.
  11. Implementation Details.
  12. Performance Evaluation.
  13. Quantitative Results
  14. Comparision with Baseline Models.
  15. The FCN Supervision Enhanced the Embeddings of 3D T1 MRI Images.
  16. ...and 4 more sections

Figures (2)

  • Figure 1: The framework of CINP. It primarily consists of a visual encoder, a visual decoder, and a network encoder.
  • Figure 2: Workflow of network prompting. In this case, the 3D T1w MRI image is classified as class 2, since the mean of $s^2_1, s^2_2,\dots, s^2_r$ is the highest average similarity.