MindCross: Fast New Subject Adaptation with Limited Data for Cross-subject Video Reconstruction from Brain Signals

TL;DR

MindCross introduces a shared-specific encoder architecture to tackle cross-subject brain decoding for video reconstruction from EEG/fMRI signals, enabling fast adaptation to new subjects with limited data. It combines a calibration phase that updates only the new subject’s encoder with a Top-K collaboration module to leverage prior subjects, and a training regime that disentangles subject-specific and invariant information using multiple losses and gradient reversal. The approach achieves competitive cross-subject performance with a single model and demonstrates efficient new-subject adaptation on EEG-DV and CC2017 benchmarks, validated by comprehensive ablations and visualizations. This work advances practical brain decoding for BCIs by reducing data requirements and speeding up personalization while maintaining high semantic fidelity in reconstructed videos.

Abstract

Reconstructing video from brain signals is an important brain decoding task. Existing brain decoding frameworks are primarily built on a subject-dependent paradigm, which requires large amounts of brain data for each subject. However, the expensive cost of collecting brain-video data causes severe data scarcity. Although some cross-subject methods being introduced, they often overfocus with subject-invariant information while neglecting subject-specific information, resulting in slow fine-tune-based adaptation strategy. To achieve fast and data-efficient new subject adaptation, we propose MindCross, a novel cross-subject framework. MindCross's N specific encoders and one shared encoder are designed to extract subject-specific and subject-invariant information, respectively. Additionally, a Top-K collaboration module is adopted to enhance new subject decoding with the knowledge learned from previous subjects' encoders. Extensive experiments on fMRI/EEG-to-video benchmarks demonstrate MindCross's efficacy and efficiency of cross-subject decoding and new subject adaptation using only one model.

Figures (13)

• Figure 1: Illustration of cross-subject brain decoding: (a) Subject-dependent paradigm: training a particular model for each subject. (b) MindCross paradigm: training a universal model for all subjects, which can fastly adapt to new subjects with limited data.
• Figure 2: The framework of proposed MindCross consisting of training, calibration, and test phase. (a) In the training phase, each specific encoder and shared encoder are optimized by several loss functions. (b) In the calibration phase, only the specific encoder of the new subject, marked with a flame icon, will be updated. (c) In the test phase, the final predictions are obtained from shared decoder and Top-K Collaborate module.
• Figure 3: Top-K Collaborate Module: The similarity vector is obtained by feeding the new subject's feature $s^t$ into the domain classifier, then we select the Top-K similar domains for collaborating to calculate the final output.
• Figure 4: Comparison on EEG-to-video benchmark.
• Figure 5: Comparison on fMRI-to-video benchmark.