Aligned with LLM: a new multi-modal training paradigm for encoding fMRI activity in visual cortex

TL;DR

The paper tackles encoding visual cortex activity from fMRI by incorporating textual semantics derived from pre-trained LLMs. It introduces the LLM-Visual Encoding Model (LLM-VEM), which generates high-quality stimulus descriptions with miniGPT-4, embeds them with CLIP text features, and trains with a contrastive image-text alignment loss to fuse multimodal information into the encoding pipeline. A two-stage training regime with PCA-based voxel mapping and an EVA02-based image extractor enables efficient learning and mitigates overfitting while aligning image features with textual representations. Experimental results on NSD/Algonauts benchmarks show notable gains and reveal that a small alignment weight $\lambda$ yields the best performance, validating the value of cross-modal supervision for brain encoding. The approach offers a practical path to integrate LLM-derived semantic priors into brain-inspired visual encoding models, potentially benefiting neuroscience research and multimodal brain-computer interfaces.

Abstract

Recently, there has been a surge in the popularity of pre trained large language models (LLMs) (such as GPT-4), sweeping across the entire Natural Language Processing (NLP) and Computer Vision (CV) communities. These LLMs have demonstrated advanced multi-modal understanding capabilities and showcased strong performance across various benchmarks. The LLM has started to embody traits of artificial general intelligence, which holds vital guidance for enhancing brain-like characteristics within visual encoding models. Hence, This paper proposes a new multi-modal training paradigm, aligning with LLM, for encoding fMRI activity in visual cortex. Based on this paradigm, we trained an encoding model in fMRI data named the LLM-Visual Encoding Model (LLM-VEM). Specifically, we utilize LLM (miniGPT4) to generate descriptive text for all stimulus images, forming a high-quality textual description set. Moreover, we use the pre-trained text encoder (CLIP) to process these detailed descriptions, obtaining the text embedding features. Next, we use the contrast loss function to minimize the distance between the image embedding features and the text embedding features to complete the alignment operation of the stimulus image and text information. With the assistance of the pre-trained LLM, this alignment process facilitates better learning of the visual encoding model, resulting in higher precision. The final experimental results indicate that our training paradigm has significantly aided in enhancing the performance of the visual encoding model.

Figures (3)

• Figure 1: The overall architecture of the proposed multi-modal training paradigm, aligning with LLM, for encoding fMRI activity in visual cortex. A represents the process of the training paradigm we designed; B illustrates the specific details of the Voxel Mapping Block in A.
• Figure 2: Prediction performance. A Vertex-wise ${{R}^{2}}$ maps for each subject on the validation split. B Individual sample images (left), fMRI targets (middle), and predictions (right) for NSD Subject 5. C Group median ${{R}^{2}}$ scores for individual ROIs.
• Figure 3: Performance results of our model in The Algonauts Project 2023 competition.