Decoupling the components of geometric understanding in Vision Language Models

TL;DR

This paper probes whether modern vision-language systems can visually grasp elementary geometric concepts when not aided by reading or reasoning. Using a cognitive-science inspired paradigm, it adapts Dehaene et al.'s Munduruku geometry stimuli and tests US adults, Munduruku adults, and Gemini Pro 1.5 on six-idea sets with an odd-one-out task, including rotation-controlled variants to probe mental rotation. The results show consistent underperformance of VLMs relative to humans, with US adults most consistent and VLMs the least, and show that mental rotation substantially reduces VLM performance but not human performance. The findings suggest different origins of geometric understanding in humans and machines and point to the potential role of formal education versus real-world interaction in acquiring geometry concepts.

Abstract

Understanding geometry relies heavily on vision. In this work, we evaluate whether state-of-the-art vision language models (VLMs) can understand simple geometric concepts. We use a paradigm from cognitive science that isolates visual understanding of simple geometry from the many other capabilities it is often conflated with such as reasoning and world knowledge. We compare model performance with human adults from the USA, as well as with prior research on human adults without formal education from an Amazonian indigenous group. We find that VLMs consistently underperform both groups of human adults, although they succeed with some concepts more than others. We also find that VLM geometric understanding is more brittle than human understanding, and is not robust when tasks require mental rotation. This work highlights interesting differences in the origin of geometric understanding in humans and machines -- e.g. from printed materials used in formal education vs. interactions with the physical world or a combination of the two -- and a small step toward understanding these differences.

Figures (6)

• Figure 1: Overview. A. Adapted from dehaene2006core to illustrate the geometric concepts we explore in this work. For the full set of stimuli refer to Appendix \ref{['fig:all_stim']}. B. An example of the experimental protocol. This is what human participants saw, VLMs received a similar prompt. See main text for details. C. Overview of results; see main text for discussion.
• Figure 2: Mental rotation. A. Stimuli inspired by the original Munduruku study aimed to measure sensitivity to chirality and metric equidistance, that implicitly assumes mental rotation. B. Rotation controlled version of the chirality and metric. C. Results on humans and Gemini on rotation controlled stimuli. The Munduruku were only evaluated for "random", and achieve approximately 60$\%$ in both the chiral and metric categories.
• Figure 3: The full set of stimuli adapted from dehaene2006core of the geometric concepts we explore in this work.
• Figure 4: Entropy of Gemini choices across categories.
• Figure 5: Accuracy across 7 vision-language models in geometric concept task.