Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts

Zekun Wang, Sashank Varma

TL;DR

The paper investigates whether geometric-topological GT concepts are innate or learned from everyday visual experience by evaluating CNNs, vision transformers, and vision-language models on a Dehaene-style odd-one-out GT task and comparing results to human data. It finds that transformer-based models show the strongest GT concept sensitivity and align most closely with children's performance, while vision-language models underperform and align less with human profiles, suggesting naive multimodal grounding may hinder abstract geometric understanding. These results support a learning-from-environment account for GT concepts in artificial systems and highlight potential limitations of current multimodal alignment approaches for abstract relations. The work provides a computational framework for testing cognitive and developmental hypotheses about GT concept acquisition and informs future model design and curricula for learning geometric reasoning.

Abstract

With the rapid improvement of machine learning (ML) models, cognitive scientists are increasingly asking about their alignment with how humans think. Here, we ask this question for computer vision models and human sensitivity to geometric and topological (GT) concepts. Under the core knowledge account, these concepts are innate and supported by dedicated neural circuitry. In this work, we investigate an alternative explanation, that GT concepts are learned ``for free'' through everyday interaction with the environment. We do so using computer visions models, which are trained on large image datasets. We build on prior studies to investigate the overall performance and human alignment of three classes of models -- convolutional neural networks (CNNs), transformer-based models, and vision-language models -- on an odd-one-out task testing 43 GT concepts spanning seven classes. Transformer-based models achieve the highest overall accuracy, surpassing that of young children. They also show strong alignment with children's performance, finding the same classes of concepts easy vs. difficult. By contrast, vision-language models underperform their vision-only counterparts and deviate further from human profiles, indicating that naïve multimodality might compromise abstract geometric sensitivity. These findings support the use of computer vision models to evaluate the sufficiency of the learning account for explaining human sensitivity to GT concepts, while also suggesting that integrating linguistic and visual representations might have unpredicted deleterious consequences.

Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts

TL;DR

The paper investigates whether geometric-topological GT concepts are innate or learned from everyday visual experience by evaluating CNNs, vision transformers, and vision-language models on a Dehaene-style odd-one-out GT task and comparing results to human data. It finds that transformer-based models show the strongest GT concept sensitivity and align most closely with children's performance, while vision-language models underperform and align less with human profiles, suggesting naive multimodal grounding may hinder abstract geometric understanding. These results support a learning-from-environment account for GT concepts in artificial systems and highlight potential limitations of current multimodal alignment approaches for abstract relations. The work provides a computational framework for testing cognitive and developmental hypotheses about GT concept acquisition and informs future model design and curricula for learning geometric reasoning.

Abstract

With the rapid improvement of machine learning (ML) models, cognitive scientists are increasingly asking about their alignment with how humans think. Here, we ask this question for computer vision models and human sensitivity to geometric and topological (GT) concepts. Under the core knowledge account, these concepts are innate and supported by dedicated neural circuitry. In this work, we investigate an alternative explanation, that GT concepts are learned ``for free'' through everyday interaction with the environment. We do so using computer visions models, which are trained on large image datasets. We build on prior studies to investigate the overall performance and human alignment of three classes of models -- convolutional neural networks (CNNs), transformer-based models, and vision-language models -- on an odd-one-out task testing 43 GT concepts spanning seven classes. Transformer-based models achieve the highest overall accuracy, surpassing that of young children. They also show strong alignment with children's performance, finding the same classes of concepts easy vs. difficult. By contrast, vision-language models underperform their vision-only counterparts and deviate further from human profiles, indicating that naïve multimodality might compromise abstract geometric sensitivity. These findings support the use of computer vision models to evaluate the sufficiency of the learning account for explaining human sensitivity to GT concepts, while also suggesting that integrating linguistic and visual representations might have unpredicted deleterious consequences.
Paper Structure (16 sections, 4 figures, 1 table)

This paper contains 16 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. GT concept understanding in humans
  4. GT concept understanding in ML models
  5. Method
  6. Models
  7. Stimuli and Datasets
  8. Evaluation on Neural Models
  9. Results
  10. Overall sensitivity
  11. Sensitivity by class
  12. Alignment to human profiles
  13. Discussion
  14. Transformer models show higher sensitivity to GT concepts
  15. Modality alignment with text reduces the sensitivity to GT concepts
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Example stimuli from the 7 categories from Dehaene2006. The odd-one-out is indicated by the red box.
  • Figure 2: Average accuracy on the odd-one-out task for both the ML models and the human participants.
  • Figure 3: Accuracy profiles of the models and humans for each of the 7 classes of GT concepts.
  • Figure 4: Heatmap of Pearson $r$ coefficients between human participants’ sensitivity profiles and the models’ sensitivity profiles across the 7 classes of GT concepts. Note that * denotes $p < 0.05$ and ** denotes $p < 0.01$.