Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Model

Zijian Dong, Yilei Wu, Zijiao Chen, Yichi Zhang, Yueming Jin, Juan Helen Zhou

TL;DR

This work tackles the problem of adapting large-scale fMRI pre-trained representations to low-resource downstream tasks, where full fine-tuning risks overfitting and distorting learned features. It introduces Scaffold Prompt Tuning (ScaPT), a parameter-efficient framework with a hierarchical prompt structure and a Deeply-conditioned Input-Prompt (DIP) mapping that maps inputs into prompt spaces while freezing the backbone. ScaPT operates in two stages—Source Training to build group-level phenotype prompts from modular prompts, and Target Training to fuse these knowledge prompts with a new target prompt via attention-guided interpolation—resulting in updates to only about 2% of parameters. Experiments on resting-state fMRI data for neurodegenerative diagnosis/prognosis and personality trait prediction show ScaPT outperforms fine-tuning and multitask prompt baselines, with interpretable attention patterns aligning with known brain-behavior relationships. Overall, ScaPT offers an efficient, interpretable path for deploying large-scale fMRI representations in low-resource clinical and behavioral neuroimaging tasks.

Abstract

We introduce Scaffold Prompt Tuning (ScaPT), a novel prompt-based framework for adapting large-scale functional magnetic resonance imaging (fMRI) pre-trained models to downstream tasks, with high parameter efficiency and improved performance compared to fine-tuning and baselines for prompt tuning. The full fine-tuning updates all pre-trained parameters, which may distort the learned feature space and lead to overfitting with limited training data which is common in fMRI fields. In contrast, we design a hierarchical prompt structure that transfers the knowledge learned from high-resource tasks to low-resource ones. This structure, equipped with a Deeply-conditioned Input-Prompt (DIP) mapping module, allows for efficient adaptation by updating only 2% of the trainable parameters. The framework enhances semantic interpretability through attention mechanisms between inputs and prompts, and it clusters prompts in the latent space in alignment with prior knowledge. Experiments on public resting state fMRI datasets reveal ScaPT outperforms fine-tuning and multitask-based prompt tuning in neurodegenerative diseases diagnosis/prognosis and personality trait prediction, even with fewer than 20 participants. It highlights ScaPT's efficiency in adapting pre-trained fMRI models to low-resource tasks.

Prompt Your Brain: Scaffold Prompt Tuning for Efficient Adaptation of fMRI Pre-trained Model

TL;DR

This work tackles the problem of adapting large-scale fMRI pre-trained representations to low-resource downstream tasks, where full fine-tuning risks overfitting and distorting learned features. It introduces Scaffold Prompt Tuning (ScaPT), a parameter-efficient framework with a hierarchical prompt structure and a Deeply-conditioned Input-Prompt (DIP) mapping that maps inputs into prompt spaces while freezing the backbone. ScaPT operates in two stages—Source Training to build group-level phenotype prompts from modular prompts, and Target Training to fuse these knowledge prompts with a new target prompt via attention-guided interpolation—resulting in updates to only about 2% of parameters. Experiments on resting-state fMRI data for neurodegenerative diagnosis/prognosis and personality trait prediction show ScaPT outperforms fine-tuning and multitask prompt baselines, with interpretable attention patterns aligning with known brain-behavior relationships. Overall, ScaPT offers an efficient, interpretable path for deploying large-scale fMRI representations in low-resource clinical and behavioral neuroimaging tasks.

Abstract

We introduce Scaffold Prompt Tuning (ScaPT), a novel prompt-based framework for adapting large-scale functional magnetic resonance imaging (fMRI) pre-trained models to downstream tasks, with high parameter efficiency and improved performance compared to fine-tuning and baselines for prompt tuning. The full fine-tuning updates all pre-trained parameters, which may distort the learned feature space and lead to overfitting with limited training data which is common in fMRI fields. In contrast, we design a hierarchical prompt structure that transfers the knowledge learned from high-resource tasks to low-resource ones. This structure, equipped with a Deeply-conditioned Input-Prompt (DIP) mapping module, allows for efficient adaptation by updating only 2% of the trainable parameters. The framework enhances semantic interpretability through attention mechanisms between inputs and prompts, and it clusters prompts in the latent space in alignment with prior knowledge. Experiments on public resting state fMRI datasets reveal ScaPT outperforms fine-tuning and multitask-based prompt tuning in neurodegenerative diseases diagnosis/prognosis and personality trait prediction, even with fewer than 20 participants. It highlights ScaPT's efficiency in adapting pre-trained fMRI models to low-resource tasks.
Paper Structure (16 sections, 4 equations, 2 figures, 2 tables)

This paper contains 16 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Setup.
  4. Overview.
  5. Deeply-conditioned Input-Prompt (DIP) Mapping.
  6. Source Training (ST).
  7. Target Training (TT).
  8. Experiments
  9. Datasets.
  10. Training Details.
  11. Main Results.
  12. Prompt Interpretation.
  13. Ablation Study and Parameter-efficiency.
  14. Conclusion
  15. Acknowledgments.
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Schematic overview of Scaffold Prompt Tuning (ScaPT) framework. ScaPT operates in two stages: Source Training (ST), where it creates phenotype prompts by interpolation of modular prompts, and Target Training (TT), where it blends phenotype prompts with a new target prompt for downstream tasks with fewer resources. Interpolation weights are determined by the attention between the input and prompts.
  • Figure 2: Further analysis: (1) visualization of 2D phenotype prompts space, (2) interpretation of target tasks through attention distribution, (3) ablation study, and (4) comparison of model performance versus numbers of parameters.