TROI: Cross-Subject Pretraining with Sparse Voxel Selection for Enhanced fMRI Visual Decoding

TL;DR

This work tackles cross-subject fMRI visual decoding by addressing the variability of brain anatomy and the noise introduced by manually labeled ROIs. It introduces Trainable Region of Interest (TROI), a two-stage framework that first learns a sparse voxel mask via Stage1 to form a compact input for a subject-specific ROI module, and then applies learning rate rewinding in Stage2 to fine-tune the input layer using limited data from a new subject. The approach combines cross-subject pretraining with MixCo-based data augmentation, CLIP-based retrieval losses, and diffusion-prior reconstruction losses, achieving improvements over annotated ROI baselines on the NSD dataset with small samples. This yields practical benefits for scalable fMRI decoding, reducing labeling burden and enabling robust cross-subject performance in data-scarce scenarios, with potential impact on brain decoding applications and neuroscience research.

Abstract

fMRI (functional Magnetic Resonance Imaging) visual decoding involves decoding the original image from brain signals elicited by visual stimuli. This often relies on manually labeled ROIs (Regions of Interest) to select brain voxels. However, these ROIs can contain redundant information and noise, reducing decoding performance. Additionally, the lack of automated ROI labeling methods hinders the practical application of fMRI visual decoding technology, especially for new subjects. This work presents TROI (Trainable Region of Interest), a novel two-stage, data-driven ROI labeling method for cross-subject fMRI decoding tasks, particularly when subject samples are limited. TROI leverages labeled ROIs in the dataset to pretrain an image decoding backbone on a cross-subject dataset, enabling efficient optimization of the input layer for new subjects without retraining the entire model from scratch. In the first stage, we introduce a voxel selection method that combines sparse mask training and low-pass filtering to quickly generate the voxel mask and determine input layer dimensions. In the second stage, we apply a learning rate rewinding strategy to fine-tune the input layer for downstream tasks. Experimental results on the same small sample dataset as the baseline method for brain visual retrieval and reconstruction tasks show that our voxel selection method surpasses the state-of-the-art method MindEye2 with an annotated ROI mask.

Figures (3)

• Figure 1: Visual reconstructions from fMRI using baseline MindEye2 w/wo our proposed TROI in different training data volume.
• Figure 2: Overview of TROI module structure followed by a residual MLP backbone and two task-specific submodules. Taking Subject 1 as a new subject, the cross-subject backbone network is trained using data from Subjects 2, 3, 5, and 7. The subject-specific TROI module is then trained in two stages with small sample data from Subject 1.
• Figure 3: TROI mask trained with the first 1h data of subject 1