Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

TL;DR

This work tackles the challenge of decoding non-invasive brain signals by introducing a Vision Transformer 3D (ViT3D)–based fMRI feature extractor that preserves three-dimensional brain structure and aligns with multi-level visual embeddings (CLIP and VAE). By employing a unified backbone, the approach enables cross-subject, single-trial visual reconstruction and seamless integration with Large Language Models (LLMs) for brain captioning, Q&A, and complex reasoning, while extending the NSD dataset with rich language annotations. The methodology combines a dual-stream fMRI extractor, multimodal interaction with LLMs, and an UnCLIP-based visual reconstruction with GradCAM-based concept localization, achieving state-of-the-art results across captioning, reconstruction, and semantic localization tasks. The work advances non-invasive brain decoding with interpretable, linguistically grounded representations and paves the way for more capable brain-computer interfaces and cognitive models, while acknowledging limitations in generalization, computational cost, and ethical considerations.

Abstract

Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition but faces challenges due to individual differences and complex neural signal representations. Traditional methods often require customized models and extensive trials, lacking interpretability in visual reconstruction tasks. Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D. This unified feature extractor efficiently aligns fMRI features with multiple levels of visual embeddings, eliminating the need for subject-specific models and allowing extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with Large Language Models (LLMs). Additionally, we have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development. Integrating with LLMs enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. Our approach demonstrates superior performance across these tasks, precisely identifying language-based concepts within brain signals, enhancing interpretability, and providing deeper insights into neural processes. These advances significantly broaden the applicability of non-invasive brain decoding in neuroscience and human-computer interaction, setting the stage for advanced brain-computer interfaces and cognitive models.

Figures (12)

• Figure 1: Overview of the integrated multimodal framework combining fMRI feature extraction with LLMs for interactive communication and reconstruction. The architecture comprises: (a) a dual-stream pathway for feature alignment with VAE and CLIP embeddings. (b) A 3D fMRI preprocessor $p$, and an fMRI feature extractor. (c) A multimodal LLM integrated with fMRI. The extracted features are then fed into an LLM for processing natural language instructions and generating responses or visual reconstructions.
• Figure 2: Description of fMRI data preprocessing. First align the data of different subjects, then patch them, and finally remove activity-irrelevant patches.
• Figure 3: Demonstration of the model's capabilities for engaging in multi-round dialogue, complex reasoning, visual reconstruction, and concept location tasks using fMRI data.
• Figure 4: Visual reconstruction results showcasing the comparison between (a) using the average signal from all trials and (b) using the first visual stimulus.
• Figure 5: Ablation analysis of the hyperparameter $\beta$ on visual reconstruction performance.