The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling

TL;DR

The paper tackles zero-shot cross-subject fMRI-to-image reconstruction by introducing UniCortex-fMRI, a unified cortical-surface dataset combining NSD, BOLD5000, NOD, and HCP-Movie, and PictorialCortex, a compositional latent framework. It learns a universal cortical latent space from UK Biobank data and factorizes fMRI latents into stimulus-driven content, subject, dataset, and nuisance components via the Latent Factorization–Composition Module (LFCM); Paired Factorization and Reconstruction and Re-Factorizing Consistency Regularization enforce robust disentanglement and generalization. During inference, surrogate latents synthesized under seen-subject conditions guide diffusion-based image synthesis to reconstruct unseen subjects’ visual experiences. Across four large datasets, PictorialCortex achieves state-of-the-art zero-shot cross-subject reconstructions, with ablations showing the critical roles of latent factorization, compositional components, and multi-dataset training. This work provides both a principled cross-subject evaluation resource and a scalable decoding framework that enhances subject-agnostic neural representations for brain–machine interfaces and cognitive neuroscience research.

Abstract

Decoding visual experiences from human brain activity remains a central challenge at the intersection of neuroscience, neuroimaging, and artificial intelligence. A critical obstacle is the inherent variability of cortical responses: neural activity elicited by the same visual stimulus differs across individuals and trials due to anatomical, functional, cognitive, and experimental factors, making fMRI-to-image reconstruction non-injective. In this paper, we tackle a challenging yet practically meaningful problem: zero-shot cross-subject fMRI-to-image reconstruction, where the visual experience of a previously unseen individual must be reconstructed without subject-specific training. To enable principled evaluation, we present a unified cortical-surface dataset -- UniCortex-fMRI, assembled from multiple visual-stimulus fMRI datasets to provide broad coverage of subjects and stimuli. Our UniCortex-fMRI is particularly processed by standardized data formats to make it possible to explore this possibility in the zero-shot scenario of cross-subject fMRI-to-image reconstruction. To tackle the modeling challenge, we propose PictorialCortex, which models fMRI activity using a compositional latent formulation that structures stimulus-driven representations under subject-, dataset-, and trial-related variability. PictorialCortex operates in a universal cortical latent space and implements this formulation through a latent factorization-composition module, reinforced by paired factorization and re-factorizing consistency regularization. During inference, surrogate latents synthesized under multiple seen-subject conditions are aggregated to guide diffusion-based image synthesis for unseen subjects. Extensive experiments show that PictorialCortex improves zero-shot cross-subject visual reconstruction, highlighting the benefits of compositional latent modeling and multi-dataset training.

Figures (14)

• Figure 1: fMRI responses of Subject 1 and Subject 2 under the same images: (a) raw fMRI responses on the cortical surface, (b) t-SNE of raw fMRI space, and (c) t-SNE of PictorialCortex stimulus-driven latent space. Colors indicate different images; star markers denote Subject 1, and upward triangles denote Subject 2. In the raw fMRI space, responses cluster by subject rather than image, while in the stimulus-driven latent space, the same images from different subjects are closely aligned.
• Figure 2: Visualizations of fMRI responses from two subjects viewing the same visual stimuli. In each row, the first column shows the ground-truth image. Columns 2--4 display three fMRI responses from Subject 1 under repeated presentations of the same image , while columns 5--7 show the corresponding three fMRI responses from Subject 2. Full size is in Appendix.
• Figure 3: (a) Visualization of the visual cortex (VC) highlighted within the whole cortical surface. (b) ROI-level parcellation within the visual cortex, where colors indicate different visual areas.
• Figure 4: Overview of the PictorialCortex framework. Given an fMRI signal arising from a visual stimulus, the observed cortical response is jointly influenced by stimulus-driven visual content as well as non-stimulus variability, including subject identity, dataset context, and trial-wise nuisance. (i) Top-left: Universal cortical representation learning maps fMRI inputs to a shared representational space, enabling reconstruction across subjects. (ii) Top-right: We first encode the fMRI input into a universal fMRI latent $\mathbf{z}$ using the universal encoder. Then, we perform compositional latent modeling via the proposed Latent Factorization--Composition Module (LFCM). The Factorizer decomposes each universal latent into a stimulus-driven code $\mathbf{c}$ and a nuisance code $\mathbf{n}$, conditioned on subject and dataset codes. These components are recombined by the Compositor to synthesize surrogate fMRI latents $\tilde{\mathbf{z}}$. A subsequent re-factorization step is trained to enforce consistency when re-encoding surrogate fMRI latents. Both the original stimulus-driven code $\mathbf{c}$ and the re-factorized stimulus-driven code $\mathbf{c}'$ are aligned with the ground-truth visual target $\mathbf{c}_{\mathrm{gt}}$, while the nuisance codes $\mathbf{n}$ and $\mathbf{n}^{\prime}$ are aligned to encourage consistency under re-factorization. (iii) Bottom: During inference, fMRI signals from unseen subjects are factorized, composited and re-factorized to produce a refined stimulus-driven code ($\mathbf{c}^{\prime}_{\mathrm{te}}$) using seen subject conditions, which conditions the diffusion model to generate the reconstructed image ($\hat{\mathbf{I}}$).
• Figure 5: Overview of Paired Factorization and Reconstruction (PFR). Paired fMRI observations elicited by the same visual stimulus are encoded into a universal latent space and decomposed into stimulus-driven and nuisance components under subject and dataset conditioning. The stimulus-driven components from the paired samples are aligned to a shared visual target. To further strengthen disentanglement, a pairwise swapping operation exchanges the stimulus-driven codes between paired observations while preserving the other factors, and consistency is enforced by reconstructing the original fMRI signals both with and without swapping. Together, PFR constrains the stimulus-driven latent to be invariant across paired observations, while allowing non-stimulus factors to account for other variability.