SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction

TL;DR

This work tackles semantic hallucination in fMRI-to-image reconstruction by recentering the problem on explicit semantic interpretation. It introduces MimeVis to generate multi-granularity, sentence-level semantic descriptions grounded to COCO captions using grounded vision–language models, and SynMind, a four-module pipeline that maps fMRI to a shared semantic and visual space to condition a diffusion-based renderer. Across NSD-based evaluations, SynMind achieves state-of-the-art semantic fidelity while using lighter backbones (Stable Diffusion 1.4) and even showing strong semantic-only performance (SynMind*). Human perceptual studies and neuroimaging analyses further show broader, semantically relevant brain engagement, supporting the practical value of semantic-guided decoding for faithful reconstruction.

Abstract

Recent advances in fMRI-based image reconstruction have achieved remarkable photo-realistic fidelity. Yet, a persistent limitation remains: while reconstructed images often appear naturalistic and holistically similar to the target stimuli, they frequently suffer from severe semantic misalignment -- salient objects are often replaced or hallucinated despite high visual quality. In this work, we address this limitation by rethinking the role of explicit semantic interpretation in fMRI decoding. We argue that existing methods rely too heavily on entangled visual embeddings which prioritize low-level appearance cues -- such as texture and global gist -- over explicit semantic identity. To overcome this, we parse fMRI signals into rich, sentence-level semantic descriptions that mirror the hierarchical and compositional nature of human visual understanding. We achieve this by leveraging grounded VLMs to generate synthetic, human-like, multi-granularity textual representations that capture object identities and spatial organization. Built upon this foundation, we propose SynMind, a framework that integrates these explicit semantic encodings with visual priors to condition a pretrained diffusion model. Extensive experiments demonstrate that SynMind outperforms state-of-the-art methods across most quantitative metrics. Notably, by offloading semantic reasoning to our text-alignment module, SynMind surpasses competing methods based on SDXL while using the much smaller Stable Diffusion 1.4 and a single consumer GPU. Large-scale human evaluations further confirm that SynMind produces reconstructions more consistent with human visual perception. Neurovisualization analyses reveal that SynMind engages broader and more semantically relevant brain regions, mitigating the over-reliance on high-level visual areas.

Figures (5)

• Figure 1: Reconstructed images produced by existing methods with publicly released results. Red boxes mark objects whose semantics are incorrectly reconstructed or misaligned with the original stimuli, while green boxes highlight semantically challenging regions that SynMind is able to reconstruct correctly.
• Figure 2: (a) Illustration of the two-round pipeline of MimeVis to prompt Qwen2-VL generate textural descriptions of an image while maximally simulating human visual perception process. In the first round, we give Qwen2-VL an open mic and encourage it to cover as many aspects as it could of an image. In the second round, we start to enforce constraints on both the lengths and contents of a finalized caption and importantly introduce COCO caption as a ground truth description to reduce synthetic hallucinations. (b) An example of the results from MimeVis. $I$ and $v_{cc}$ are visual stimuli and its original COCO caption, while $v_{N}$ represents outputs of MimeVis under different lengths limits $N$.
• Figure 3: Overview of the proposed fMRI-to-image reconstruction framework, SynMind. The framework consists of four core modules: the Subject-Wise Mapper (SWM), the Subject-Shared Semantic Encoder (SSE), the Subject-Shared Visual Encoder (SSV), and the Semantic-Aware Renderer (SAR). In the figure, black dashed arrows ($\dashrightarrow$) indicate operations used only during training, while gray dashed arrows ($\dashrightarrow$) denote optional components that can be enabled or disabled at inference time. Solid gray arrows ($\rightarrow$) illustrate the essential semantic-guided workflow from fMRI signals to image reconstruction.
• Figure 4: Qualitative comparison of fMRI-based image reconstructions between SynMind and existing state-of-the-art methods. Towards fair comparison, whenever possible, we directly replicated their reconstruction results from their published papers. In cases where such results are not available, we regenerate the results using their publicly released model weights.
• Figure 5: Visualization of ROI importance for Subjects 01, 02, 05, and 07, derived from the learned weights of the first layer in the Subject-Wise Mapper (SWM) of SynMind*. For each subject, voxel-wise weights are first averaged across all modules and then normalized such that the total activation weight across voxels sums to 1, where red and yellow correspond to lower and higher importance, respectively. $N=\{30,45,60,75\}$ indicates different levels of granularity of the intermediate semantic features extracted by MimeVis. White text labels denote ROI names and are displayed at fixed reference locations for visualization clarity rather than exact subject-specific ROI centroids.