H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

TL;DR

This work delivers a robust, large-scale estimation of human performance on the ARC benchmark by evaluating 1,729 participants on the full original task set (400 training, 400 evaluation). It reports training and evaluation accuracies of roughly 76% and 64%, respectively, with pessimistic/optimistic bounds, and demonstrates that virtually all tasks are solvable by at least one person, highlighting a substantial human–AI gap. The authors publicly release H-ARC, a dataset containing all submissions and action traces to accelerate progress in human-like abstract reasoning and visual program synthesis. Through extensive analysis of performance, timing, and error types, the paper sheds light on how people solve ARC problems and where current AI approaches diverge from human reasoning.

Abstract

The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test challenging out-of-distribution generalization in humans and machines. Since 2019, limited progress has been observed on the challenge using existing artificial intelligence methods. Comparing human and machine performance is important for the validity of the benchmark. While previous work explored how well humans can solve tasks from the ARC benchmark, they either did so using only a subset of tasks from the original dataset, or from variants of ARC, and therefore only provided a tentative estimate of human performance. In this work, we obtain a more robust estimate of human performance by evaluating 1729 humans on the full set of 400 training and 400 evaluation tasks from the original ARC problem set. We estimate that average human performance lies between 73.3% and 77.2% correct with a reported empirical average of 76.2% on the training set, and between 55.9% and 68.9% correct with a reported empirical average of 64.2% on the public evaluation set. However, we also find that 790 out of the 800 tasks were solvable by at least one person in three attempts, suggesting that the vast majority of the publicly available ARC tasks are in principle solvable by typical crowd-workers recruited over the internet. Notably, while these numbers are slightly lower than earlier estimates, human performance still greatly exceeds current state-of-the-art approaches for solving ARC. To facilitate research on ARC, we publicly release our dataset, called H-ARC (human-ARC), which includes all of the submissions and action traces from human participants.

Figures (7)

• Figure 1: ARC Demonstration Tasks. Shown above are examples of easy (nearly everyone solved them in two attempts or less) and hard (almost no one solved them in two attempts or less) tasks with corresponding training examples and test for each. Interestingly, the easy evaluation task shown here has not been solved by the top LLM solutions to ARC reported in this document. Below the tasks are state space graphs representing all visited states by participants, from starting state (blue nodes) to correct or incorrect submitted grid (green and red nodes respectively). From left to right: f76d97a5.json., e9ac8c9e.json, e3497940.json and dd2401ed.json.
• Figure 2: ARC Experiment Interface. Participants were given instructions about the different controls and layout of the interface followed by a tutorial task. Shown here is the evaluation tutorial task e9afcf9a.json.
• Figure 3: Mean 3-shot task success rate. Tasks are ordered from lowest success rate to highest, showing the distribution of empirically estimated task difficulty for both the 400 tasks in the training and evaluation sets. Dotted lines show average accuracy across all tasks in either the training (blue) or evaluation (orange) split.
• Figure 4: Human behavior action traces on ARC problems. In the left column, we show the test input seen by participants and the true test output grid for three different problems from the ARC evaluation set. In the middle column, sampled action traces show successive states of the grid with the last state corresponding to a correct (green box) or incorrect (red box) submission. In the last column, we show the first natural language descriptions submitted by participants along with their solution. From top to bottom: 34b99a2b.json, 4364c1c4.json and a8610ef7.json.
• Figure 5: Minimum number of correct submissions across tasks. In this figure, we report the proportion of tasks where a minimum of N (x-axis) participants submitted a correct solution in three attempts or less. Note that higher values of N are biased negatively since posterior inference on a binomial outcome with small sample size is skewed.