MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

TL;DR

MindEye2 tackles the practical barrier of subject-specific, data-hungry fMRI-to-image models by introducing a shared-subject framework pretrained across multiple subjects and fine-tuned on as little as 1 hour of data from a new subject. The method maps fMRI activity to a rich CLIP image-embedding space via a linear shared-subject alignment, a residual backbone, and a diffusion prior, with image reconstruction performed by fine-tuning Stable Diffusion XL in unCLIP mode and refining outputs through caption guidance and low-level submodules. It achieves state-of-the-art reconstruction and retrieval metrics on the Natural Scenes Dataset and demonstrates strong 1-hour data performance, validated by human preferences and ablation analyses. The work suggests a practical path for deploying fMRI-based vision reconstructions in clinical or brain-computer interface settings, while acknowledging limitations related to movement sensitivity and image distribution scope.

Abstract

Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on GitHub.

Figures (13)

• Figure 1: MindEye2 vs. MindEye1 reconstructions from fMRI brain activity using varying amounts of training data.
• Figure 2: MindEye2 overall schematic. MindEye2 is trained using samples from 7 subjects in the Natural Scenes Dataset and then fine-tuned using a target held-out subject who may have scarce training data. Ridge regression maps fMRI activity to an initial shared-subject latent space. An MLP backbone and diffusion prior output OpenCLIP ViT-bigG/14 embeddings which SDXL unCLIP uses to reconstruct the seen image, which are then refined with base SDXL. The submodules help retain low-level information and support retrieval tasks. Snowflakes=frozen models used during inference, flames=actively trained.
• Figure 3: SDXL unCLIP reconstructions + predicted image captions (left) are fed to base SDXL for refinement (right).
• Figure 4: Reconstructions from different model approaches using 1 hour of training data from NSD.
• Figure 5: Normalized reconstruction metrics for MindEye2 with (connected) or without (dotted) pretraining on other subjects, using varying amounts of training/fine-tuning data. Normalization was such that $0$ on the y-axis corresponds to metrics using random COCO images (not from NSD test set) as reconstructions and $1$ corresponds to metrics using 40-session pretrained MindEye2. Black lines indicate median. Test data is the same across all comparisons (see section \ref{['results']}).