MindShot: A Few-Shot Brain Decoding Framework via Transferring Cross-Subject Prior and Distilling Frequency Domain Knowledge

TL;DR

MindShot addresses the challenge of decoding visual stimuli from fMRI with limited subject-specific data by transferring cross-subject priors and distilling frequency-domain knowledge. It introduces MSP to encode cross-subject semantic priors and an FKD module to reduce inter-subject variability in the frequency domain, enabling effective few-shot adaptation and diffusion-based image reconstruction guided by CLIP semantics. The approach achieves high semantic fidelity on NSD with dramatically reduced scan time, e.g., attaining 83.6% CLIP accuracy with only 1.8% of the data and surpassing several PSPM baselines in few-shot settings. This work suggests a practical path toward scalable, data-efficient brain decoding suitable for clinical and personalized neuroimaging applications, supported by publicly available code.

Abstract

Aiming to reconstruct visual stimuli from brain signals, brain decoding has recently made significant progress using functional magnetic resonance imaging (fMRI). However, it still has challenging issues such as substantial individual differences and high data collection costs. To simplify these problems, most methods adopt the per-subject-per-model paradigm, but this greatly limits their applications. In this paper, we design a few-shot brain decoding setting specifically for potential clinical scenarios and propose a novel two-stage decoding framework named MindShot, comprising a Multi-Subject Pretraining (MSP) stage and Fourier-based cross-subject Knowledge Distillation (FKD) stage. Firstly, a MSP framework based on multi-modal contrastive learning is constructed to mine the cross-subject prior. Secondly, the FKD is presented to decrease inter-individual differences while improving the decoding adaptability to new individuals. Our approach achieves high semantic fidelity in visual reconstruction on the largest dataset and has the potential to reduce scanning time by up to 99%. Remarkably, MindShot achieves a CLIP accuracy of 83.6% using only 1.8% of the fMRI-image pairs, surpassing the 77.4% accuracy of the method trained on the entire NSD dataset. This makes it feasible to train large-scale brain decoding frameworks that require less data, facilitating practical applications. The code is available at https://github.com/JSinBUPT/MindShot.

Figures (11)

• Figure 1: Comparison with existing brain decoding methods. (a) Current fMRI-to-image methods rely on training PSPM paradigms with approximately 40 hours of scan data. (b) These methods suffer from significant performance degradation when only limited data is available. (c) Our MindShot demonstrates improved performance by effectively leveraging cross-subject prior knowledge in few-shot scenarios.
• Figure 2: The overall architecture of MindShot. MindShot is a two-stage framework designed for few-shot fMRI-to-image brain decoding. (a) In the first stage, multi-subject prior knowledge pretraining is acquired through a multi-modal contrastive learning using the $\mathcal{L}_{semantic}$, which leads to the pretrained $\mathcal{E}_{brain}$. (b) In the second stage, under the few-shot setting, we propose a Fourier-based cross-subject Knowledge Distillation (FKD) module to eliminate individual differences in the frequency domain, thereby transferring the prior knowledge from $\mathcal{E}_{brain}$ and improving image reconstruction based on diffusion model. (c) The FKD module eliminates individual differences by aligning the dimensions of new and pretrained subjects using a Normalizer first, and then applying an Eliminator to adjust the signal distribution of the new subject, resulting in the signal $\hat{x}_i$. where the $\mathcal{L}_{Fourier}$ is the supervision in the frequency domain. This is achieved by transforming the signals to the frequency domain using FFT, and then minimizing the distances between $\mathcal{A}_i$, $\mathcal{A}_j$, $\mathcal{P}_i$, and $\mathcal{P}_j$.
• Figure 3: Quantitative comparison with state-of-the-art methods under few-shot settings. Our method reconstructs images with both semantically meaningful content and fine-grained details.
• Figure 4: Visualization of one-shot brain decoding across multiple subjects, with correct recovery of food, sports, vehicles, and indoor scenes achieved under 0.36h scan time.
• Figure 5: Brain decoding performance comparison of MindShot under two sampling methods: few-shot and random sampling. The dashed line represents the average score across all scan times.