MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data

TL;DR

MindAligner tackles cross-subject visual decoding from scarce fMRI data by introducing an explicit functional alignment framework. It learns a Brain Transfer Matrix $oldsymbol{oldsymbol{ abla M}}$ decomposed as $oldsymbol{oldsymbol{ abla M}} = oldsymbol{oldsymbol{ abla A}} imes oldsymbol{oldsymbol{ abla B}}$ and trains a Brain Functional Alignment module with a Cross-Stimulus Neural Mapper to establish fine-grained voxel-level correspondences via multi-level losses $oldsymbol{oldsymbol{ abla L}}_{ ext{rec}}$, $oldsymbol{oldsymbol{ abla L}}_{ ext{KL}}$, and $oldsymbol{oldsymbol{ abla L}}_{ ext{latent}}$. During inference only the BTM is used, enabling cross-subject decoding with limited data and efficient parameter usage (~6% of MindEye2). Empirical results on NSD show robust improvements in fMRI-to-image reconstruction and reveal neuroscience insights about region-specific inter-subject variability, while maintaining a lightweight, interpretable alignment framework.

Abstract

Brain decoding aims to reconstruct visual perception of human subject from fMRI signals, which is crucial for understanding brain's perception mechanisms. Existing methods are confined to the single-subject paradigm due to substantial brain variability, which leads to weak generalization across individuals and incurs high training costs, exacerbated by limited availability of fMRI data. To address these challenges, we propose MindAligner, an explicit functional alignment framework for cross-subject brain decoding from limited fMRI data. The proposed MindAligner enjoys several merits. First, we learn a Brain Transfer Matrix (BTM) that projects the brain signals of an arbitrary new subject to one of the known subjects, enabling seamless use of pre-trained decoding models. Second, to facilitate reliable BTM learning, a Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli with a multi-level brain alignment loss, uncovering fine-grained functional correspondences with high interpretability. Experiments indicate that MindAligner not only outperforms existing methods in visual decoding under data-limited conditions, but also provides valuable neuroscience insights in cross-subject functional analysis. The code will be made publicly available.

Figures (8)

• Figure 1: Different approaches to functional alignment in brain decoding: Prior works mindeyev2eccv25 adopt implicit alignment approach that aligns all subjects into a single latent space, which may lead to suboptimal alignment. Differently, MindAligner employs an explicit alignment strategy, mapping novel subject signals to seen ones by establishing fine-grained functional correspondences. MindAligner not only enables high-quality visual reconstruction from fMRI signals but also facilitates brain functional analysis across subjects.
• Figure 2: Overview of MindAligner. To achieve explicit brain functional alignment, given a pre-trained brain decoding model, we design a Brain Functional Alignment Module (BFA) that learns a Brain Transfer Matrix (BTM) $\boldsymbol{\mathcal{M}}$ for fMRI mapping between the known and novel subjects. BTM is decomposed into two low-rank matrices $\boldsymbol{\mathcal{A}}$ and $\boldsymbol{\mathcal{B}}$ to create latent space for further alignment. The Cross-Stimulus Neural Mapper is proposed to create fMRI pairs under shared stimuli. In addition to the alignment losses $\boldsymbol{\mathcal{L}}_{rec}$ and $\boldsymbol{\mathcal{L}}_{KL}$ between generated and real fMRI, a latent alignment loss $\boldsymbol{\mathcal{L}}_{latent}$ guides functional alignment based on stimulus similarities. In the inference stage, only the BTM is utilized for functional mapping, enabling cross-subject brain decoding.
• Figure 3: Visualization of MindAligner's decoding results from training on one hour of data.
• Figure 4: Visualization results of aligning a new subject with different known subjects.
• Figure 5: Visualization of transfer quantity in brain heatmaps.