BrainMCLIP: Brain Image Decoding with Multi-Layer feature Fusion of CLIP

TL;DR

BrainMCLIP tackles fMRI-to-image decoding by introducing a neuro-inspired, parameter-efficient fusion of CLIP features across multiple layers. By mapping low- and high-level brain signals to intermediate and final CLIP layers and incorporating a cross-reconstruction mechanism plus a multi-granularity loss, it achieves strong semantic decoding while preserving fine visual detail without a VAE. The approach yields competitive or superior high-level semantic performance with a 71.7% reduction in parameters compared to VAE-based state-of-the-art methods, and aligns predicted features with brain hierarchies as shown on NSD. This work demonstrates that leveraging CLIP's intermediate representations, in tandem with brain-inspired processing, can enhance fMRI-based image reconstruction with greater efficiency and detail fidelity, offering a scalable path for next-generation brain decoding.

Abstract

Decoding images from fMRI often involves mapping brain activity to CLIP's final semantic layer. To capture finer visual details, many approaches add a parameter-intensive VAE-based pipeline. However, these approaches overlook rich object information within CLIP's intermediate layers and contradicts the brain's functionally hierarchical. We introduce BrainMCLIP, which pioneers a parameter-efficient, multi-layer fusion approach guided by human visual system's functional hierarchy, eliminating the need for such a separate VAE pathway. BrainMCLIP aligns fMRI signals from functionally distinct visual areas (low-/high-level) to corresponding intermediate and final CLIP layers, respecting functional hierarchy. We further introduce a Cross-Reconstruction strategy and a novel multi-granularity loss. Results show BrainMCLIP achieves highly competitive performance, particularly excelling on high-level semantic metrics where it matches or surpasses SOTA(state-of-the-art) methods, including those using VAE pipelines. Crucially, it achieves this with substantially fewer parameters, demonstrating a reduction of 71.7\%(Table.\ref{tab:compare_clip_vae}) compared to top VAE-based SOTA methods, by avoiding the VAE pathway. By leveraging intermediate CLIP features, it effectively captures visual details often missed by CLIP-only approaches, striking a compelling balance between semantic accuracy and detail fidelity without requiring a separate VAE pipeline.

Figures (5)

• Figure 1: Reconstructions guided by different CLIP vision layers reveal a clear trade-off. Intermediate layers capture fine details but introduce semantic noise (red boxes), while the final layer ensures semantic consistency at the cost of detail accuracy (blue boxes). Our proposed fusion of intermediate (layers 10-20) and final layers ('Fused') achieves a compelling balance between detail fidelity and semantic coherence. More results are shown in Appendix.A.
• Figure 2: (Top) Similarity between fMRI features from different visual functional regions (Section. fMRI Data Processing) of subject 01 and CLIP vision model layers. (Bottom) Aggregated high-level visual regions are compared with low-level regions, showing their fMRI feature similarities against CLIP layers. Results for other subjects are provided in Appendix.B.
• Figure 3: Overview of BrainMCLIP. BrainMCLIP consists of Text and Image branches, both employing an MLP-based backbone and fMRI-to-image decoders. The Text branch aligns fMRI features with the final layer features of the CLIP Text model, while the Image branch aligns them with the fused features from the CLIP Vision model. A Cross-reconstruction module in the Image branch prevents noise learning and improves performance.
• Figure 4: The heatmap illustrating the Centered Kernel Alignment (CKA) between intermediate layer features of the CLIP vision model.
• Figure 5: The image reconstruction results of BrainMCLIP. We show more results in Appendix.F.