MindBridge: A Cross-Subject Brain Decoding Framework

TL;DR

MindBridge tackles the core limitation of subject-specific brain decoding by introducing a single cross-subject model that unifies heterogeneous fMRI inputs via an adaptive aggregation function and learns subject-invariant semantic embeddings through a cycle-based reconstruction mechanism. It integrates a diffusion-based image synthesis backbone guided by CLIP image and text embeddings, enabling faithful image reconstruction and even novel fMRI signal synthesis across subjects. A novel reset-tuning adaptation strategy allows efficient transfer to new subjects with limited data, aided by cycle-consistent pseudo data augmentation. Empirical results on the NSD dataset show competitive cross-subject decoding performance, strong new-subject adaptation with limited data, and meaningful fMRI synthesis capabilities, underscoring potential for broader neuroscience applications and more data-efficient brain decoding.

Abstract

Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle reconstruction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Furthermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page: https://littlepure2333.github.io/MindBridge

Figures (7)

• Figure 1: Image stimuli and images reconstructed from captured brain signals. Given the limited training data from a new subject (subj07 from the NSD dataset allen2022massive), our proposed MindBridge can faithfully reconstruct natural images using less data, benefiting from pretrained cross-subject knowledge. In contrast, the Vanilla method, which represents current methods following a per-subject-per-model paradigm, fails to learn effectively from limited data.
• Figure 2: Overview of MindBridge. MindBridge is a cross-subject brain decoding framework capable of handling fMRI signals from different subjects. Initially, an aggregation function unifies the size of fMRI signals. Subsequently, subject-wise brain embedders and brain builders are trained to obtain subject-invariant semantic embeddings. The Brain Translator then generates text and image embeddings, which are utilized to reconstruct images through versatile diffusion model. The dimension of data is denoted within the box.
• Figure 3: Brain decoding results with only one model. Unlike previous methods, which confine one model to a specific subject, our proposed cross-subject brain decoding framework, MindBridge, can reconstruct images from multiple subjects using just one model.
• Figure 4: Novel fMRI synthesis within MindBridge pretrained on subject 1, 2, 5. The fMRI signals of subjects 5 and 2 are converted into subject 1’s fMRI signals through cycle reconstruction, then subject 1’s brain embedder are utilized for brain decoding.
• Figure 5: More cross-subject reconstructions of MindBridge on subject 1, 2, 5 and 7.