Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding

Yudan Ren, Xinlong Wang, Kexin Wang, Tian Xia, Zihan Ma, Zhaowei Li, Xiangrong Bi, Xiao Li, Xiaowei He

TL;DR

This study investigates brain-like processing in vision-language transformers by deriving fine-grained artificial neurons from attention modules and encoding their dynamics against fMRI signals. Using a two-stage pipeline with sparse dictionary learning to extract 128 temporal patterns and voxel-wise LASSO encoding, the authors compare CLIP and METER against unimodal baselines (ViT, RoBERTa) on movie-derived brain data from 184 participants. They find that ANs predict biological neurons across multiple networks, reveal functional redundancy and polarity alignments, and show architecture-driven differences in brain-like integration—CLIP tending toward modality-specific pathways and METER toward cross-modal fusion. These results support brain-like hierarchical processing in VLMs and offer neuroscience-informed insights for multimodal AI design and evaluation, particularly regarding cross-modal integration mechanisms.

Abstract

While brain-inspired artificial intelligence(AI) has demonstrated promising results, current understanding of the parallels between artificial neural networks (ANNs) and human brain processing remains limited: (1) unimodal ANN studies fail to capture the brain's inherent multimodal processing capabilities, and (2) multimodal ANN research primarily focuses on high-level model outputs, neglecting the crucial role of individual neurons. To address these limitations, we propose a novel neuron-level analysis framework that investigates the multimodal information processing mechanisms in vision-language models (VLMs) through the lens of human brain activity. Our approach uniquely combines fine-grained artificial neuron (AN) analysis with fMRI-based voxel encoding to examine two architecturally distinct VLMs: CLIP and METER. Our analysis reveals four key findings: (1) ANs successfully predict biological neurons (BNs) activities across multiple functional networks (including language, vision, attention, and default mode), demonstrating shared representational mechanisms; (2) Both ANs and BNs demonstrate functional redundancy through overlapping neural representations, mirroring the brain's fault-tolerant and collaborative information processing mechanisms; (3) ANs exhibit polarity patterns that parallel the BNs, with oppositely activated BNs showing mirrored activation trends across VLM layers, reflecting the complexity and bidirectional nature of neural information processing; (4) The architectures of CLIP and METER drive distinct BNs: CLIP's independent branches show modality-specific specialization, whereas METER's cross-modal design yields unified cross-modal activation, highlighting the architecture's influence on ANN brain-like properties. These results provide compelling evidence for brain-like hierarchical processing in VLMs at the neuronal level.

Uncovering Brain-Like Hierarchical Patterns in Vision-Language Models through fMRI-Based Neural Encoding

TL;DR

This study investigates brain-like processing in vision-language transformers by deriving fine-grained artificial neurons from attention modules and encoding their dynamics against fMRI signals. Using a two-stage pipeline with sparse dictionary learning to extract 128 temporal patterns and voxel-wise LASSO encoding, the authors compare CLIP and METER against unimodal baselines (ViT, RoBERTa) on movie-derived brain data from 184 participants. They find that ANs predict biological neurons across multiple networks, reveal functional redundancy and polarity alignments, and show architecture-driven differences in brain-like integration—CLIP tending toward modality-specific pathways and METER toward cross-modal fusion. These results support brain-like hierarchical processing in VLMs and offer neuroscience-informed insights for multimodal AI design and evaluation, particularly regarding cross-modal integration mechanisms.

Abstract

While brain-inspired artificial intelligence(AI) has demonstrated promising results, current understanding of the parallels between artificial neural networks (ANNs) and human brain processing remains limited: (1) unimodal ANN studies fail to capture the brain's inherent multimodal processing capabilities, and (2) multimodal ANN research primarily focuses on high-level model outputs, neglecting the crucial role of individual neurons. To address these limitations, we propose a novel neuron-level analysis framework that investigates the multimodal information processing mechanisms in vision-language models (VLMs) through the lens of human brain activity. Our approach uniquely combines fine-grained artificial neuron (AN) analysis with fMRI-based voxel encoding to examine two architecturally distinct VLMs: CLIP and METER. Our analysis reveals four key findings: (1) ANs successfully predict biological neurons (BNs) activities across multiple functional networks (including language, vision, attention, and default mode), demonstrating shared representational mechanisms; (2) Both ANs and BNs demonstrate functional redundancy through overlapping neural representations, mirroring the brain's fault-tolerant and collaborative information processing mechanisms; (3) ANs exhibit polarity patterns that parallel the BNs, with oppositely activated BNs showing mirrored activation trends across VLM layers, reflecting the complexity and bidirectional nature of neural information processing; (4) The architectures of CLIP and METER drive distinct BNs: CLIP's independent branches show modality-specific specialization, whereas METER's cross-modal design yields unified cross-modal activation, highlighting the architecture's influence on ANN brain-like properties. These results provide compelling evidence for brain-like hierarchical processing in VLMs at the neuronal level.
Paper Structure (18 sections, 5 equations, 7 figures)

This paper contains 18 sections, 5 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Consistency between ANNs and Human Brain Neuron response
  4. Neuron Representation Mapping with Multimodal Transformer Models
  5. Method
  6. Overview
  7. ANs and Their Temporal Response in VLMs
  8. Representative Temporal Response Patterns of ANs
  9. Voxel-wise Encoding of Brain Activity
  10. Experiments
  11. Dataset and Preprocessing
  12. Implementation Details
  13. Results
  14. Applicability of the Sparse Representation in Artificial and Biological Neuron
  15. Mapping between Sparse Dictionary and Brain Regions
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Overview of the study. (a) Data Acquisition. (b) Construction of Fine-Grained ANs from Multi-Head Self-Attention Modules. (c) Extraction of Representative Activation Patterns of ANs via SDL. (d) Predicting BNs using Voxel-wise Encoding Model.
  • Figure 2: Illustration of the AN construction process.
  • Figure 3: The probability distribution of $R^{2}$ for different branches of models in ANs and BNs. (a)-(d), (j)-(k): $R^{2}$ distribution for ANs. (e)-(h), (i)-(m): $R^{2}$ distribution for BNs.
  • Figure 4: Brain activation maps corresponding to example dictionary atoms. Color intensity represents the t-statistic from the group-level one-sample t-test on the LASSO coefficients ($A_{fmri}$ in Eq. \ref{['eq:encoding']}) for that atom across subjects, thresholded for significance (FDR corrected p < 0.05). Warm colors indicate positive coefficients (atom positively predicts activity); cool colors indicate negative coefficients. Maps are displayed on an inflated cortical surface. Examples shown activate regions including the auditory cortex (a-d), visual cortex (e-h), attention networks (e-h), default mode network (i-l), and language-related areas (i-l).
  • Figure 5: Number of each functional parcellation in 128 BNs. 128 is the number of dictionary derived from ANs.
  • ...and 2 more figures