ARC Is a Vision Problem!

TL;DR

This work reframes the Abstraction and Reasoning Corpus (ARC) as a vision-centric image-to-image translation problem called Vision ARC (VARC). By placing inputs on a flexible canvas, employing a Vision Transformer (with 2D positional embeddings and patch-based tokens), and applying translation/scale priors plus test-time training, VARC achieves 54.5% ARC-1 accuracy with a single model (and 60.4% when ensembling), approaching average human performance. The approach emphasizes visual priors, scalable architecture choices, and multi-view inference to enable strong cross-task generalization from ARC data alone. Altogether, VARC demonstrates that abstract reasoning tasks in ARC can be effectively tackled through vision-centric representations and learning, suggesting broader applicability of image-based priors for reasoning.

Abstract

The Abstraction and Reasoning Corpus (ARC) is designed to promote research on abstract reasoning, a fundamental aspect of human intelligence. Common approaches to ARC treat it as a language-oriented problem, addressed by large language models (LLMs) or recurrent reasoning models. However, although the puzzle-like tasks in ARC are inherently visual, existing research has rarely approached the problem from a vision-centric perspective. In this work, we formulate ARC within a vision paradigm, framing it as an image-to-image translation problem. To incorporate visual priors, we represent the inputs on a "canvas" that can be processed like natural images. It is then natural for us to apply standard vision architectures, such as a vanilla Vision Transformer (ViT), to perform image-to-image mapping. Our model is trained from scratch solely on ARC data and generalizes to unseen tasks through test-time training. Our framework, termed Vision ARC (VARC), achieves 60.4% accuracy on the ARC-1 benchmark, substantially outperforming existing methods that are also trained from scratch. Our results are competitive with those of leading LLMs and close the gap to average human performance.

Figures (22)

• Figure 1: The ARC benchmark (top) consists of a collection of many different tasks, where each task has a few (e.g., 2-4) examples. We propose the Vision ARC (VARC) framework, which addresses the ARC problem as an image-to-image translation problem, from a computer vision perspective (bottom). In this illustration, the underlying concepts of the three tasks can be roughly described by humans as: "reflection" (left), "symmetry" (middle), and "gravity" (right). These concepts are closely related to the visual and physical world.
• Figure 2: Examples of unseen tasks solved by VARC. Each panel shows an unseen test task, with demonstrations on the top and the model’s prediction on the bottom. VARC correctly solves these challenging tasks.
• Figure 3: The ARC problem definition. ARC is a collection of many different tasks. For each task, a few (e.g., 2-4) demonstration pairs $(x, y)$ are given, and the model is required to infer the output from $x_\textrm{infer}$. The training set $\mathcal{T}_\textrm{train}$ is a collection of 400 tasks, which can be used for model training. The test set $\mathcal{T}_\textrm{test}$ contains 400 new tasks: the demo pairs of a new task are given only at inference time, based on which the model performs inference on $x_\textrm{infer}$.
• Figure 4: The raw input undergoes random scale and translation transformations and is placed on the "canvas" (denoted in gray).
• Figure 5: The ViT architecture in VARC. The input is randomly placed on a canvas, which is then treated as a natural image and processed by a standard ViT, conditioned on the task token.