Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI

TL;DR

This work investigates reconstructing visual stimuli from fMRI and finds that the text space of language models is the most brain-aligned latent space for bridging neural activity to image generation. Building on this, it introduces PRISM, which maps fMRI signals into a structured text space and uses an object-centric diffusion generator guided by automatically discovered object attributes and spatial relationships. The approach integrates an attribute–relationship search with per-object encoders and a diffusion model that composes objects to form images, achieving up to 8% lower perceptual loss and improved perceptual fidelity across NSD, BOLD5000, and GOD datasets. The results underscore the value of structured text as an intermediate representation for brain-to-image reconstruction and demonstrate practical gains in reconstructing semantically meaningful, object-level visual content from neural activity.

Abstract

Understanding how the brain encodes visual information is a central challenge in neuroscience and machine learning. A promising approach is to reconstruct visual stimuli, essentially images, from functional Magnetic Resonance Imaging (fMRI) signals. This involves two stages: transforming fMRI signals into a latent space and then using a pretrained generative model to reconstruct images. The reconstruction quality depends on how similar the latent space is to the structure of neural activity and how well the generative model produces images from that space. Yet, it remains unclear which type of latent space best supports this transformation and how it should be organized to represent visual stimuli effectively. We present two key findings. First, fMRI signals are more similar to the text space of a language model than to either a vision based space or a joint text image space. Second, text representations and the generative model should be adapted to capture the compositional nature of visual stimuli, including objects, their detailed attributes, and relationships. Building on these insights, we propose PRISM, a model that Projects fMRI sIgnals into a Structured text space as an interMediate representation for visual stimuli reconstruction. It includes an object centric diffusion module that generates images by composing individual objects to reduce object detection errors, and an attribute relationship search module that automatically identifies key attributes and relationships that best align with the neural activity. Extensive experiments on real world datasets demonstrate that our framework outperforms existing methods, achieving up to an 8% reduction in perceptual loss. These results highlight the importance of using structured text as the intermediate space to bridge fMRI signals and image reconstruction.

Figures (5)

• Figure 1: Framework Overview: PRISM generates structured text descriptions for each training image using a VLM to iteratively extract brain-aligned object attributes and relationships. These descriptions capture the image’s compositional and relational content and serve as supervision to train an encoder and fine-tune a language model to map fMRI signals into the text space. During inference, the model predicts descriptions from fMRI signals, which then guide a pre-trained diffusion model for object-centric image reconstruction.
• Figure 2: Reconstructed images from different methods. The first column shows the original viewed images. The rest of the columns show the reconstructed images from different methods.
• Figure 3: A Common Error in Generative Models. While the original image shows “a gray tiger-striped cat,” the model incorrectly generates “a grey tiger,” illustrating semantic distortion.
• Figure 4: Scores of all tested relation keywords during the prompt search. Among the top-10 relation keywords, the most frequent keyword is 'spatial'.
• Figure :