FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning

TL;DR

This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.

Abstract

Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.

Figures (6)

• Figure 1: The model architecture of our proposed FCN-LLM that supports queries related to various FCN-involved tasks. It employs multi-scale encoder to extract FCN features, maps these features into the text embedding space.
• Figure 2: Illustration of the proposed multi-paradigm task design for instruction tuning (left), and the multi-stage learning of FCN knowledge learning for our proposed FCN-LLM (right).
• Figure 3: (a) The impact of continual training on the generalization performance of FCN-LLM in Stage2; (b) The ablation study on the proposed multi-scale FCN encoder, which explores the effects of different types of FCN tokens—specifically, combinations of 116 ROI-level tokens, 7 subnetwork-level tokens, and 1 brain-level token.
• Figure 4: Visualization of connections among FCN tokens of different groups of diseases from the subnetwork perspective.
• Figure 5: Visualization of statistical evidence comprehensive.