Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)

TL;DR

The work asks whether instruction-tuned multimodal LLMs can better align with human brain activity and whether their instruction-specific embeddings capture distinct neural representations. Using NSD fMRI data, the authors compare three MLLMs (InstructBLIP, mPLUG-Owl, IDEFICS) against ViT-H and CLIP across 10 instructions and six visual tasks, employing voxelwise ridge encoders and variance partitioning. They find that MLLMs outperform vision-only models and approach CLIP in brain alignment, with IC prompting yielding strongest alignment in high-level visual regions (EBA, PPA, FFA) and other prompts biasing early visual areas; most variance across instructions is shared, suggesting room to sharpen instruction differentiation. Layer-wise analyses show middle layers often align with higher visual areas while later layers align with early visual processing, highlighting model architecture differences. Overall, instruction-tuning enhances multimodal brain alignment and reveals instruction-specific neural encoding patterns, offering guidelines for improving brain-predictive representations.

Abstract

Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models-through increased size, instruction-tuning, and multimodality-has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions show that MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results suggest that enhancing MLLMs' ability to capture task-specific information could lead to better differentiation between various types of instructions, and thereby improving their precision in predicting brain responses.

Figures (22)

• Figure 1: Leveraging instruction-tuned multimodal LLMs for brain encoding with a diverse set of instructions. For the given image, we could obtain different multimodal representations using instructions that ask the model to (i) generate the caption of the image, (ii) identify whether people are present, or (iii) determine the primary colors dominant in the image. Using instruction-specific representations, we estimate the alignment using a simple linear function $f$ (ridge regression) which map MLLM representations to brain recordings.
• Figure 2: Whole visual cortex and ROI-based normalized brain alignment was computed by averaging across participants, layers, and voxels. Blue: Average across random initialization of the 3 MLLMs. Note that CLIP-text model uses golden oracle captions while instruct models use predicted model generations. $\ast$ indicates cases where MLLM embeddings are statistically significantly better than randomly initialized models, i.e., p$\leq 0.05$. $\wedge$ indicates cases where MLLMs are significantly better than unimodal vision models (ViT-H), i.e., p$\leq 0.05$. Other brain ROI plots are reported in Fig. \ref{['fig:other_localizers_brain_nsd']} in Appendix \ref{['app:visual_functional_localizers']}.
• Figure 3: Each voxel is color coded with the instruction (out of 10) that led to the highest normalized brain alignment. The color bar highlights color codes for each instruction. The voxels are projected onto the flattened cortical surface of a representative subject (subject S1) for two MLLMs. Similar brain maps for other subjects are in Appendix \ref{['app:subjectSpecificMapsTaskSpecific']}.
• Figure 4: Voxels specific to visual concepts groups: Counts (Left) and Recognition (Right). The color bar from Fig. \ref{['fig:instruction_pycortex_nsd']} highlights color codes for each instruction. The voxels are projected onto the flattened cortical surface of a representative subject (subject S1).
• Figure 5: Each voxel is color coded with the MLLM layer number (out of 33) that led to the highest normalized brain alignment. The color bar highlights color codes for each layer. The voxels are projected onto the flattened cortical surface of a representative subject (subject S1) for three MLLMs. Similar brain maps for other subjects are in Appendix \ref{['app:subjectSpecificMapsLayerSpecific']}.