Do Large Language Models Solve ARC Visual Analogies Like People Do?

TL;DR

This study compares human and large language model (LLM) performance on simplified ARC-style visual analogies using two KidsARC datasets designed to probe different developmental stages. It employs a one-shot learning setup, collects human data in a museum, and surveys a broad set of LLMs (including GPT-3/4/4V) with standardized prompts, followed by a structured error taxonomy to compare solution strategies. Humans generally outperform LLMs, with LLMs prone to copying inputs or employing simple matrix-based combinations, while humans show more concept-based errors and deeper abstractions when ambiguous items allow multiple valid solutions. The findings suggest LLMs currently rely on surface statistics rather than robust abstract representations, and they highlight the value of error analyses and carefully designed ambiguous items for diagnosing and guiding the development of more relationally capable AI systems. The work also discusses limitations due to uncontrolled data collection and calls for controlled replication and development of architectures that support symbol-like abstractions and compositional reasoning in visual tasks.

Abstract

The Abstraction Reasoning Corpus (ARC) is a visual analogical reasoning test designed for humans and machines (Chollet, 2019). We compared human and large language model (LLM) performance on a new child-friendly set of ARC items. Results show that both children and adults outperform most LLMs on these tasks. Error analysis revealed a similar "fallback" solution strategy in LLMs and young children, where part of the analogy is simply copied. In addition, we found two other error types, one based on seemingly grasping key concepts (e.g., Inside-Outside) and the other based on simple combinations of analogy input matrices. On the whole, "concept" errors were more common in humans, and "matrix" errors were more common in LLMs. This study sheds new light on LLM reasoning ability and the extent to which we can use error analyses and comparisons with human development to understand how LLMs solve visual analogies.

Figures (5)

• Figure 1: Example item and interface from KidsARC-Simple task (top). Corresponding prompt given to LLMs (bottom), derived from Moskvichev23.
• Figure 2: Examples of different error types. See Figure \ref{['fig:exampleitem']} for the full item. A Copy error (partially) duplicates an input matrix. A Matrix error combines inputs. Concept errors apply correct transformations or concepts, but a mistake is made (here the wrong color). Other represents idiosyncratic responses.
• Figure 3: Item-wise comparison of human vs. LLM performance on KidsARC-Simple and KidsARC-Concept. The items are visualized on the left of the performance bars. On the right we display the three most common responses (models and humans) along with percentage occurrence. Green ticks indicate correct responses. Note: items can have more than one correct response.
• Figure 4: As with young children, copying is a common error in LLMs. Compared to humans, matrix-based errors are more common in LLMs, while concept-based errors are less common. Other errors are common in both LLMs and young children.
• Figure 5: Comparison of accuracy and error types model-by-model and for humans. We plot models with different training regimes - Base Models, Fine-tuned Models, Mixture of Experts, GPTs, as well as Human participants. Note: To reduce clutter we do not plot 'other', empty, or invalid responses, hence some bars do not sum up to eight.