Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding

TL;DR

The paper tackles the challenge of multi-subject brain visual decoding, which is hindered by anatomical differences and subject-specific fMRI patterns. It introduces Wills Aligner, a framework combining anatomical alignment to a standard template, Mixture of Brain Experts adapters with a global router, and a two-phase meta-learning strategy that leverages semantic relations to share decoding knowledge across subjects. The approach yields state-of-the-art results across classification, retrieval, and image reconstruction on the NSD dataset, with strong few-shot performance and robust ablations showing the value of each component. This work offers a practical path toward scalable, cross-subject brain decoding, enabling more generalizable brain-computer interface applications and neuroscience research.

Abstract

Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, the diversity in cortical parcellation and fMRI patterns across individuals has prompted the development of deep learning models tailored to each subject. The personalization limits the broader applicability of brain visual decoding in real-world scenarios. To address this issue, we introduce Wills Aligner, a novel approach designed to achieve multi-subject collaborative brain visual decoding. Wills Aligner begins by aligning the fMRI data from different subjects at the anatomical level. It then employs delicate mixture-of-brain-expert adapters and a meta-learning strategy to account for individual fMRI pattern differences. Additionally, Wills Aligner leverages the semantic relation of visual stimuli to guide the learning of inter-subject commonality, enabling visual decoding for each subject to draw insights from other subjects' data. We rigorously evaluate our Wills Aligner across various visual decoding tasks, including classification, cross-modal retrieval, and image reconstruction. The experimental results demonstrate that Wills Aligner achieves promising performance.

Figures (10)

• Figure 1: Illustrations of inter-subject brain differences. The left figure illustrates structural differences, showing variations in size and location of the same functional areas (highlighted in the same color) across subjects. The right figure depicts differences in fMRI patterns, where distinct brain activities are observed when the same stimulus is presented.
• Figure 2: Overview of the proposed Wills Aligner. Figure (a) illustrates the pipeline of our method. Figure (b) shows its three key components: Anatomical Alignment, Mixture of Brain Experts, and Learning Strategy.
• Figure 3: Reconstruction results for Subj01.
• Figure 4: Analysis on router. The left figure shows the growth of the router's test set classification accuracy with training steps. The right figure shows the average probabilities of the router's top-1 prediction on the test set.
• Figure 5: Retrieval results of Subj01 on the test set.