Efficient Multi Subject Visual Reconstruction from fMRI Using Aligned Representations

TL;DR

The paper tackles the challenge of reconstructing visual stimuli from fMRI across multiple subjects with limited data. It introduces Adapter Alignment (AA), a two-stage training paradigm that explicitly aligns subject embeddings to a reference in a common visual representation space, aided by a data-efficient greedy image selection strategy with a submodular objective and a $(1-1/e)$ approximation guarantee. AA delivers faster convergence and comparable performance to end-to-end training with full data, while substantially outperforming baselines in low-data regimes, as demonstrated on NSD and THINGS datasets and across MindEye1/MindEye2 architectures. The work shows that cross-subject fMRI reconstructions can be made data-efficient and broadly generalizable, reducing data and compute requirements and broadening practical access to brain-to-image reconstruction.

Abstract

This work introduces a novel approach to fMRI-based visual image reconstruction using a subject-agnostic common representation space. We show that the brain signals of the subjects can be aligned in this common space during training to form a semantically aligned common brain. This is leveraged to demonstrate that aligning subject-specific lightweight modules to a reference subject is significantly more efficient than traditional end-to-end training methods. Our approach excels in low-data scenarios. We evaluate our methods on different datasets, demonstrating that the common space is subject and dataset-agnostic.

Figures (17)

• Figure 1: Cross-subject cosine similarity of shared image embeddings in the common space extracted from a pre-trained MindEye2 model. (a) Alignment between subjects is weak before any transformation. (b) Applying an orthogonal transformation substantially improves cross-subject similarity. (c) Explicit alignment through lightweight adapter modules achieves near-perfect alignment. Only Subjects 1, 2, 5, and 7 are shown for consistency throughout the manuscript.
• Figure 2: Training procedures for fMRI-Image Reconstruction. Top: Traditional Approach to reconstruction. All subjects are trained simultaneously and loss functions are applied at the MLP Mapper to align the output to pre-trained CLIP space. Bottom: Proposed AdapterAlign training pipeline. First, we train a single reference subject and obtain its embeddings in the shared space. Then we train subsequent subjects to minimize loss at both the shared space and the mapper. The same mapper weights are transferred over in step 2.
• Figure 3: Pushing the limits of data efficiency with AAMax+Image Selection. Left: Subject 2 fine-tuned from Subject 1. Right: Subject 5 fine-tuned from Subject 1. In both cases, AAMax and AAMax+Image Selection outperform the baseline trained with 250 images, even when using only 150 images (a 40% reduction) and match performance with 100 images (a 60% reduction).
• Figure 4: Performance of AAMax compared to baseline fine-tuning as a function of training data size. Left: AlexNet-5 2-way percent correct. Middle: Inception 2-way percent correct. Right: CLIP 2-way percent correct. Across all three metrics, AAMax achieves higher accuracy in limited-data regimes, with the advantage diminishing as more data becomes available.
• Figure 5: Examples of reconstructed images from Subject 1 of the THINGS dataset after fine-tuning. Each pair depicts the ground truth (left) and the reconstructed image (right).

Theorems & Definitions (4)

• Definition K.1: Bin-Mapping Problem
• Definition K.2: Set Cover Problem
• Theorem K.3
• proof