Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction

TL;DR

MindHier reframes fMRI-to-image reconstruction as a coarse-to-fine process that exploits a hierarchy of neural embeddings rather than a fixed high-level feature. It introduces a Hierarchical fMRI Encoder (HFE) that maps fMRI signals to a sequence of embeddings $E=\{\mathbf{e}_1, \dots, \mathbf{e}_M\}$ and aligns them to CLIP features via a structural loss $\mathcal{L}_{MSE}$ and a semantic SoftCLIP loss. The reconstruction is performed by a scale-wise autoregressive generator with scale-aware guidance, where $p(R|E)=\prod_{k=1}^K p(r_k|r_{<k},\mathbf{s}_k)$ and $h_k= M - \left\lfloor M(k-1)/K \right\rfloor$, starting with $s_1=\mathbf{e}_M$ and progressively using $\mathbf{s}_k=\mathbf{e}_{h_k}$. On NSD, MindHier achieves state-of-the-art semantic fidelity with CLIP-based scores reaching $96.4\%$ and competitive structural metrics, while producing a $2.64$-second inference per image—about a $4.67\times$ speedup over diffusion baselines—and more deterministic reconstructions than prior methods. This cognitively aligned framework thus advances efficient, semantically faithful decoding of visual stimuli from brain activity.

Abstract

Reconstructing visual stimuli from fMRI signals is a central challenge bridging machine learning and neuroscience. Recent diffusion-based methods typically map fMRI activity to a single high-level embedding, using it as fixed guidance throughout the entire generation process. However, this fixed guidance collapses hierarchical neural information and is misaligned with the stage-dependent demands of image reconstruction. In response, we propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling. MindHier introduces three components: a Hierarchical fMRI Encoder to extract multi-level neural embeddings, a Hierarchy-to-Hierarchy Alignment scheme to enforce layer-wise correspondence with CLIP features, and a Scale-Aware Coarse-to-Fine Neural Guidance strategy to inject these embeddings into autoregression at matching scales. These designs make MindHier an efficient and cognitively-aligned alternative to diffusion-based methods by enabling a hierarchical reconstruction process that synthesizes global semantics before refining local details, akin to human visual perception. Extensive experiments on the NSD dataset show that MindHier achieves superior semantic fidelity, 4.67x faster inference, and more deterministic results than the diffusion-based baselines.

Figures (8)

• Figure 1: Comparison of fMRI-to-image reconstruction pipelines. (a) Prior diffusion-based methods utilize a fixed neural feature to guide the reconstruction. (b) In contrast, MindHier employs scale-aware guidance, leveraging hierarchical neural features to first establish a low-resolution overview ("Forest") before progressively refining local details ("Trees") at higher resolutions.
• Figure 2: An overview of the MindHier framework. (a) Hierarchy-to-Hierarchy Alignment. A hierarchical fMRI encoder, composed of $M$ cascaded transformer blocks, is trained to map an fMRI signal to a hierarchy of neural features in CLIP embedding space. This mapping is learned by aligning the encoder's outputs with corresponding intermediate features from a frozen CLIP vision encoder using a cascaded MSE loss ($\mathcal{L}_{\text{MSE}}$ (Eq. \ref{['eq:MSE']})). To ensure high-level semantic coherence, the terminal fMRI feature is also aligned within CLIP's shared embedding space via a SoftCLIP loss ($\mathcal{L}_{\text{SoftCLIP}}$ (Eq. \ref{['eq:softclip_loss']})). (b) Scale-Aware Coarse-to-Fine Neural Guidance. A scale-wise autoregressive model generates the image across $K$ scales, from a coarse, low-resolution view to high-resolution details. This generation process is conditioned on the hierarchical fMRI features from stage (a), which are incorporated using a scale-aware guidance strategy. In practice, an attention mask is used to selectively route features via cross attention, which directs the features for the coarse view to guide the initial scale and the features for finer details to guide subsequent scales. For illustration, a simplified case where $M\!=\!2$, the fMRI feature dimension $C_{\text{fMRI}}\!=\!2$, and $K\!=\!2$ is shown.
• Figure 3: Qualitative comparison of fMRI-to-image reconstructions. The stimuli include a diverse range of classes, from animals and objects to complex indoor and outdoor scenes. All data shown is from Subject 1.
• Figure 4: Comparison of reconstruction consistency across four trials. MindHier demonstrates stable and consistent outputs, in contrast to the highly variable results from the stochastic MindBridge model.
• Figure 5: Qualitative results of brain grounding. For visualization, a bounding box generated from each reconstruction is overlaid on the original visual stimulus, highlighting the precise localization of the target.