Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probing the Representational Geometry of Color Qualia: Dissociating Pure Perception from Task Demands in Brains and AI Models

Jing Xu

TL;DR

The study tackles how color qualia representations map onto neural codes and how task demands alter these representations. It combines RSA-based brain-model comparisons using a no-report vs report paradigm, a large set of >30 vision models, and a Brain-Score–driven layer-selection, yielding RSMs of size $9×9$ and a noise-ceiling bound for interpretation. Key findings show that most models align better with pure perception than with task-modulated processing, that CLIP training interacts with architecture (beneficial for ViT but detrimental for ConvNet), and that AI models develop abstract HSV-space symmetries absent in the brain, revealing divergent inductive biases. This work provides a color-qualia benchmark compatible with Brain-Score and offers clear guidance for designing more neurally plausible vision systems that bridge perception and task-driven representations.

Abstract

Probing the computational underpinnings of subjective experience, or qualia, remains a central challenge in cognitive neuroscience. This project tackles this question by performing a rigorous comparison of the representational geometry of color qualia between state-of-the-art AI models and the human brain. Using a unique fMRI dataset with a "no-report" paradigm, we use Representational Similarity Analysis (RSA) to compare diverse vision models against neural activity under two conditions: pure perception ("no-report") and task-modulated perception ("report"). Our analysis yields three principal findings. First, nearly all models align better with neural representations of pure perception, suggesting that the cognitive processes involved in task execution are not captured by current feedforward architectures. Second, our analysis reveals a critical interaction between training paradigm and architecture, challenging the simple assumption that Contrastive Language-Image Pre-training(CLIP) training universally improves neural plausibility. In our direct comparison, this multi-modal training method enhanced brain-alignment for a vision transformer(ViT), yet had the opposite effect on a ConvNet. Our work contributes a new benchmark task for color qualia to the field, packaged in a Brain-Score compatible format. This benchmark reveals a fundamental divergence in the inductive biases of artificial and biological vision systems, offering clear guidance for developing more neurally plausible models.

Probing the Representational Geometry of Color Qualia: Dissociating Pure Perception from Task Demands in Brains and AI Models

TL;DR

The study tackles how color qualia representations map onto neural codes and how task demands alter these representations. It combines RSA-based brain-model comparisons using a no-report vs report paradigm, a large set of >30 vision models, and a Brain-Score–driven layer-selection, yielding RSMs of size and a noise-ceiling bound for interpretation. Key findings show that most models align better with pure perception than with task-modulated processing, that CLIP training interacts with architecture (beneficial for ViT but detrimental for ConvNet), and that AI models develop abstract HSV-space symmetries absent in the brain, revealing divergent inductive biases. This work provides a color-qualia benchmark compatible with Brain-Score and offers clear guidance for designing more neurally plausible vision systems that bridge perception and task-driven representations.

Abstract

Probing the computational underpinnings of subjective experience, or qualia, remains a central challenge in cognitive neuroscience. This project tackles this question by performing a rigorous comparison of the representational geometry of color qualia between state-of-the-art AI models and the human brain. Using a unique fMRI dataset with a "no-report" paradigm, we use Representational Similarity Analysis (RSA) to compare diverse vision models against neural activity under two conditions: pure perception ("no-report") and task-modulated perception ("report"). Our analysis yields three principal findings. First, nearly all models align better with neural representations of pure perception, suggesting that the cognitive processes involved in task execution are not captured by current feedforward architectures. Second, our analysis reveals a critical interaction between training paradigm and architecture, challenging the simple assumption that Contrastive Language-Image Pre-training(CLIP) training universally improves neural plausibility. In our direct comparison, this multi-modal training method enhanced brain-alignment for a vision transformer(ViT), yet had the opposite effect on a ConvNet. Our work contributes a new benchmark task for color qualia to the field, packaged in a Brain-Score compatible format. This benchmark reveals a fundamental divergence in the inductive biases of artificial and biological vision systems, offering clear guidance for developing more neurally plausible models.
Paper Structure (11 sections, 3 figures)

This paper contains 11 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Human Neuroimaging Data and Experimental Design
  4. AI Vision Models
  5. Representational Similarity Analysis
  6. Result
  7. AI Models Align Better with Pure Perception
  8. Training Paradigms Interact with Model Architecture
  9. AI Models Learn Abstract Symmetries Absent in Human Brain Representations
  10. Discussion
  11. First Appendix

Figures (3)

  • Figure 1: Quantitative comparison of AI models and human brain representations.(a) Model similarity scores to neural data under no-report (x-axis) vs. report (y-axis) conditions. The dashed line represents identity ($y=x$). Nearly all models fall below this line. (b) Direct comparison of neural alignment for models with and without CLIP training.
  • Figure 2: Qualitative divergence in representational geometry. RDMs for human brain activity and AI models. The small color suqares means what color is compared. The two leftmost panels show the average RDM from human fMRI data in the no-report and report conditions, respectively. The green array, with no dark purple going through, reveals they lack any clear off-diagonal structure. In contrast, the RDM of the best-performing AI model (third panel) exhibits an anti-diagonal structure(blue array), reflecting the abstract mathematical symmetry of the HSV color space.
  • Figure 3: RDMs for all tested models. This figure shows the $9 \times 9$ RDM for each of the AI models evaluated in this study. A prominent anti-diagonal structure, highlighted by the cyan curves on select plots, is visible in many of the high-performing models, particularly those based on ViT and CLIP training paradigms. This provides visual evidence that the tendency to learn the abstract mathematical symmetry of the stimulus set is a general phenomenon across a wide range of modern vision models.