Lite-Mind: Towards Efficient and Robust Brain Representation Network

TL;DR

Lite-Mind introduces a light, robust brain representation learning paradigm for fMRI-to-image retrieval by replacing MindEye's large MLP backbone with a Discrete Fourier Transform (DFT) based backbone. The DFT Backbone employs Spectrum Compression and a Frequency Projector (FreMLP) to learn informative voxel embeddings in the frequency domain, enabling high-precision retrieval while drastically reducing parameters. A Diffusion Projector and contrastive learning align voxel embeddings with CLIP image embeddings, extending capabilities to LAION-5B retrieval and zero-shot GOD classification. The approach achieves 94.6% NSD Subject 1 retrieval with 98.7% fewer parameters than MindEye, demonstrates robustness on smaller datasets, and sets new state-of-the-art results for zero-shot GOD classification, highlighting practical implications for edge deployment and scalable brain decoding.

Abstract

The limited data availability and the low signal-to-noise ratio of fMRI signals lead to the challenging task of fMRI-to-image retrieval. State-of-the-art MindEye remarkably improves fMRI-to-image retrieval performance by leveraging a large model, i.e., a 996M MLP Backbone per subject, to align fMRI embeddings to the final hidden layer of CLIP's Vision Transformer (ViT). However, significant individual variations exist among subjects, even under identical experimental setups, mandating the training of large subject-specific models. The substantial parameters pose significant challenges in deploying fMRI decoding on practical devices. To this end, we propose Lite-Mind, a lightweight, efficient, and robust brain representation learning paradigm based on Discrete Fourier Transform (DFT), which efficiently aligns fMRI voxels to fine-grained information of CLIP. We elaborately design a DFT backbone with Spectrum Compression and Frequency Projector modules to learn informative and robust voxel embeddings. Our experiments demonstrate that Lite-Mind achieves an impressive 94.6% fMRI-to-image retrieval accuracy on the NSD dataset for Subject 1, with 98.7% fewer parameters than MindEye. Lite-Mind is also proven to be able to be migrated to smaller fMRI datasets and establishes a new state-of-the-art for zero-shot classification on the GOD dataset.

Figures (13)

• Figure 1: The significance of lightweight brain decoding model. Subject-specific models are visually represented by clouds of distinct colors, and the cloud size correlates with model parameters.
• Figure 2: Overview of our Lite-Mind. Figures (a) and (b) show the architecture of the MLP Backbone of MindEye and the DFT backbone and Retrieval Pipeline, respectively. fMRI voxels are inputted into DFT Backbone to obtain voxel embeddings.
• Figure 3: Partial retrieval results of Lite-Mind on all 982 test images for Subject 1. With 12.5M DFT Backbone, Lite-Mind can still find the exact Top-1 image pair from the test set of 982 images with 89.3% accuracy and 95.5% accuracy of image-fMRI retrieval(random chance = 0.1%) and can distinguish among confusable candidates. The number below each image represents the similarity score. See more cases including failure retrieval in Appendix C.4.
• Figure 4: Retrieval results of Lite-Mind from LAION-5B for Subject 1. The left column marked by a red box in every two columns represents the original image seen by the Subject, and the right column represents the image retrieved from LAION-5B.
• Figure 5: T-SNE of embeddings of 982 test fMRI by diffusion projector on LAION-5B retrieval for Subject 1. In the figure, image-clip represents image-clip CLS embeddings, while voxel-fft and voxel-diffusion represent voxel CLS embeddings by DFT Backbone or Diffusion Projector. The diffusion projector plays a role in bringing vectors closer, enabling image-to-image retrieval on the LAION-5B dataset.