Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Models See in Line with Human Vision? Probing the Correspondence Between LVLM Representations and EEG Signals

Xin Xiao, Yang Lei, Haoyang Zeng, Xiao Sun, Xinyi Jiang, Yu Tian, Hao Wu, Kaiwen Wei, Jiang Zhong

TL;DR

This paper quantifies LVLM-brain alignment using image-evoked Electroencephalogram (EEG) signals, analyzing the effects of model architecture, scale, and image type to demonstrate that LVLMs learn human-aligned visual representations and establish neural alignment as a biologically grounded benchmark for evaluating and improving LVLMs.

Abstract

Large Vision Language Models (LVLMs) exhibit strong visual understanding and reasoning abilities. However, whether their internal representations reflect human visual cognition is still under-explored. In this paper, we address this by quantifying LVLM-brain alignment using image-evoked Electroencephalogram (EEG) signals, analyzing the effects of model architecture, scale, and image type. Specifically, by using ridge regression and representational similarity analysis, we compare visual representations from 32 open-source LVLMs with corresponding EEG responses. We observe a structured LVLM-brain correspondence: First, intermediate layers (8-16) show peak alignment with EEG activity in the 100-300 ms window, consistent with hierarchical human visual processing. Secondly, multimodal architectural design contributes 3.4 more to brain alignment than parameter scaling, and models with stronger downstream visual performance exhibit higher EEG similarity. Thirdly, spatiotemporal patterns further align with known cortical visual pathways. These results demonstrate that LVLMs learn human-aligned visual representations and establish neural alignment as a biologically grounded benchmark for evaluating and improving LVLMs. In addition, those results could provide insights that may inform the development of neuro-inspired applications.

Do Models See in Line with Human Vision? Probing the Correspondence Between LVLM Representations and EEG Signals

TL;DR

This paper quantifies LVLM-brain alignment using image-evoked Electroencephalogram (EEG) signals, analyzing the effects of model architecture, scale, and image type to demonstrate that LVLMs learn human-aligned visual representations and establish neural alignment as a biologically grounded benchmark for evaluating and improving LVLMs.

Abstract

Large Vision Language Models (LVLMs) exhibit strong visual understanding and reasoning abilities. However, whether their internal representations reflect human visual cognition is still under-explored. In this paper, we address this by quantifying LVLM-brain alignment using image-evoked Electroencephalogram (EEG) signals, analyzing the effects of model architecture, scale, and image type. Specifically, by using ridge regression and representational similarity analysis, we compare visual representations from 32 open-source LVLMs with corresponding EEG responses. We observe a structured LVLM-brain correspondence: First, intermediate layers (8-16) show peak alignment with EEG activity in the 100-300 ms window, consistent with hierarchical human visual processing. Secondly, multimodal architectural design contributes 3.4 more to brain alignment than parameter scaling, and models with stronger downstream visual performance exhibit higher EEG similarity. Thirdly, spatiotemporal patterns further align with known cortical visual pathways. These results demonstrate that LVLMs learn human-aligned visual representations and establish neural alignment as a biologically grounded benchmark for evaluating and improving LVLMs. In addition, those results could provide insights that may inform the development of neuro-inspired applications.
Paper Structure (42 sections, 3 equations, 9 figures, 3 tables)

This paper contains 42 sections, 3 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Foundations of Vision Processing
  4. LVLM-Brain Alignment
  5. Methodology
  6. Brain Data and Preprocessing
  7. Extraction of Image Embeddings
  8. Mapping Image Embeddings to EEG Responses
  9. Multidimensional Similarity Assessment
  10. Predictive Performance Evaluation.
  11. Spatio-temporal Pattern of Predictions.
  12. Layer-wise LVLM-EEG Temporal Alignment.
  13. Category-dependent Similarity.
  14. Experiments
  15. Implementation Details
  16. ...and 27 more sections

Figures (9)

  • Figure 1: The motivation of this paper: exploring the similarity of human brain EEG signals and LVLM representations from various aspects.
  • Figure 2: Illustration of the analysis pipeline for exploring representational similarity between human EEG signals and LVLMs. Image representations from different LVLM layers are compared with EEG responses elicited by the same visual stimuli to assess model--brain alignment.
  • Figure 3: Layer-wise correspondence between the LVLMs and EEG signals across multiple participants.
  • Figure 4: Spatiotemporal dynamics of LVLM-EEG similarity.
  • Figure 5: Category-level correlations, where MI is musical instrument and GF is geological formation.
  • ...and 4 more figures