Don't throw the baby out with the bathwater: How and why deep learning for ARC

TL;DR

This work tackles the Abstraction and Reasoning Corpus (ARC) challenge by arguing that deep learning can excel in unfamiliar domains when coupled with test-time adaptation. It introduces two key techniques—Test-Time Fine Tuning (TTFT) and Augment Inference Reverse-Augmentation and Vote (AIRV)—and relies on a LongT5 encoder-decoder backbone trained with multi-task data and code-rich pretraining to enable robust, on-the-fly abstraction learning. The approach achieves large empirical gains, up to about 260% from AIRV alone and an additional ~300% when combined with TTFT, and attains a top private test-set score (58%) on ARC-AGI, signaling strong generalization to unseen riddles. The work highlights the importance of contextualized learning, test-time reframing, and perceptual reasoning priors in enabling robust ARC solvers, while offering a scalable, computation-aware path forward for cross-domain abstract reasoning.

Abstract

The Abstraction and Reasoning Corpus (ARC-AGI) presents a formidable challenge for AI systems. Despite the typically low performance on ARC, the deep learning paradigm remains the most effective known strategy for generating skillful (state-of-the-art) neural networks (NN) across varied modalities and tasks in vision, language etc. The deep learning paradigm has proven to be able to train these skillful neural networks and learn the abstractions needed in these diverse domains. Our work doubles down on that and continues to leverage this paradigm by incorporating on-the-fly NN training at test time. We demonstrate that fully committing to deep learning's capacity to acquire novel abstractions yields state-of-the-art performance on ARC. Specifically, we treat both the neural network and the optimizer (rather than just a pre-trained network) as integral components of the inference process, fostering generalization to unseen tasks. Concretely, we propose a methodology for training on ARC, starting from pretrained LLMs, and enhancing their ARC reasoning. We also propose Test-Time Fine-Tuning (TTFT) and the Augment Inference Reverse-Augmentation and Vote (AIRV) as effective test-time techniques. We are the first to propose and show deep learning can be used effectively for ARC, showing boosts of up to 260% in accuracy with AIRV and a further 300% boost with TTFT. An early version of this approach secured first place in the 2023 ARCathon competition, while the final version achieved the current best score on the ARC private test-set (58%). Our findings highlight the key ingredients of a robust reasoning system in unfamiliar domains, underscoring the central mechanisms that improve broad perceptual reasoning.

Figures (6)

• Figure 1: An example of a single easy ARC task (one datapoint). This task is solved by surrounding the red pixels with four yellow corners, blue pixels with four orange side pixels, whereas cyan and magenta input pixels remain unchanged.
• Figure 2: An example of the text prompt fed to our model. Here, each input and output grid is unrolled into a flat sequence of pixel-color values, which are then concatenated with keywords such as train, test, input, and output. The phrase solve: indicates that the model should produce the correct transformed grid (toutput1) corresponding to the given test grid (tinput1).
• Figure 3: An example of a simpler ARC riddle.
• Figure 4: AIRV process applied to a simple ARC riddle. Starting from the original riddle (blue panel), the pipeline (1) Augments the grids via rotations and flips, (2) runs inference on each transformed instance, (3) reverses every prediction back to the original frame of reference, and finally (4) votes on the most consistent output.
• Figure 5: Results for the fully trained base model in the 3 different test-time configurations.