Reasoning is a Modality

TL;DR

This work argues that reasoning constitutes a distinct modality and introduces a role-separated transformer with a small global controller and a large local workspace to solve ARC tasks. By constraining global information flow and enabling iterative rule execution through a recurrent controller, the approach achieves state-of-the-art ARC-1 performance under the VARC protocol, surpassing average human accuracy and outperforming prior methods. The model also provides qualitative evidence of more structured rule application via attention patterns, aligning with the hypothesis that a readable internal state underpins human-like reasoning. The study highlights the importance of grounding AI reasoning in explicit internal controllers and demonstrates how test-time adaptation can bolster generalization on abstract visual reasoning tasks.

Abstract

The Abstraction and Reasoning Corpus (ARC) provides a compact laboratory for studying abstract reasoning, an ability central to human intelligence. Modern AI systems, including LLMs and ViTs, largely operate as sequence-of-behavior prediction machines: they match observable behaviors by modeling token statistics without a persistent, readable mental state. This creates a gap with human-like behavior: humans can explain an action by decoding internal state, while AI systems can produce fluent post-hoc rationalizations that are not grounded in such a state. We hypothesize that reasoning is a modality: reasoning should exist as a distinct channel separate from the low-level workspace on which rules are applied. To test this hypothesis, on solving ARC tasks as a visual reasoning problem, we designed a novel role-separated transformer block that splits global controller tokens from grid workspace tokens, enabling iterative rule execution. Trained and evaluated within the VARC vision-centric protocol, our method achieved 62.6% accuracy on ARC-1, surpassing average human performance (60.2%) and outperforming prior methods significantly. Qualitatively, our models exhibit more coherent rule-application structure than the dense ViT baseline, consistent with a shift away from plausible probability blobs toward controller-driven reasoning.

Figures (6)

• Figure 1: Solving an ARC puzzle by 2 types of reasoning: amortized fitting and controller-driven reasoning. Column (a) is the ARC puzzle input-output pairs. Column (b) has two predictions by the ViT based VARC. Column (c) contains two predictions by our new model. This example shows ViT predicts the output as plausible probability blobs, whereas our model has more structured rule application. See Figure \ref{['fig_6']} for corresponding layer-wise attention maps.
• Figure 2: Workflow of our new reasoning model. We adopt the VARC method for input transform, visual embedding, and output projection. Details of our new transformer block are shown in Figure \ref{['fig_3']}. The transformer blocks can be stacked up.
• Figure 3: The internal structure of the new role-splitting transformer block. (a) and (b) show the attention mechanism of the patch token and controller token, and (c) shows the block structure.
• Figure 4: Four architecture variants of our new transformer block for separating global controller from local workspace.
• Figure 5: Ablation study and best models: training and evaluation accuracy curves across 100 training epochs for selected architectures of our new transformer block in pretraining. (a) and (b) compare the results of 5 basic architectures; while (c) and (d) show the results of 3 best models.