ARCTraj: A Dataset and Benchmark of Human Reasoning Trajectories for Abstract Problem Solving

TL;DR

ARCTraj introduces a large-scale dataset of temporally ordered, object-centric reasoning trajectories solving ARC tasks, enabling direct study of how human reasoning unfolds beyond static input–output pairs. It defines a unified pipeline and a formal MDP-compatible representation, and adds three auxiliary knowledge streams—Selection bias, Color provenance, and Shared intentions—to guide a range of learning paradigms from RL to diffusion and sequence modeling. The paper demonstrates that ARCTraj facilitates improved solver performance across multiple architectures and provides rich analyses of human strategies, including attention patterns and strategy convergence, with practical benefits for explainability and generalization. By serving as both a dataset and a modeling framework, ARCTraj advances human-centered AI research in abstract reasoning and offers a blueprint for future trajectory-based cognitive supervision across domains.

Abstract

We present ARCTraj, a dataset and methodological framework for modeling human reasoning through complex visual tasks in the Abstraction and Reasoning Corpus (ARC). While ARC has inspired extensive research on abstract reasoning, most existing approaches rely on static input--output supervision, which limits insight into how reasoning unfolds over time. ARCTraj addresses this gap by recording temporally ordered, object-level actions that capture how humans iteratively transform inputs into outputs, revealing intermediate reasoning steps that conventional datasets overlook. Collected via the O2ARC web interface, it contains around 10,000 trajectories annotated with task identifiers, timestamps, and success labels across 400 training tasks from the ARC-AGI-1 benchmark. It further defines a unified reasoning pipeline encompassing data collection, action abstraction, Markov decision process (MDP) formulation, and downstream learning, enabling integration with reinforcement learning, generative modeling, and sequence modeling methods such as PPO, World Models, GFlowNets, Diffusion agents, and Decision Transformers. Analyses of spatial selection, color attribution, and strategic convergence highlight the structure and diversity of human reasoning. Together, these contributions position ARCTraj as a structured and interpretable foundation for studying human-like reasoning, advancing explainability, alignment, and generalizable intelligence.

Figures (13)

• Figure 1: Overview of the ARCTraj data collection process. Users solve ARC tasks through the O2ARC platform by interacting with grid-based objects. Their actions are recorded step-by-step to form semantically rich, temporally ordered trajectories.
• Figure 2: Example of a single trajectory log in ARCTraj. Each action in the "actionSequence" specifies its category and operation, along with the associated grid and object state, forming a structured state–action unit.
• Figure 3: Overview of how ARCTraj analyses inform ARC solving. The ARC solver $f_\theta$ predicts $\hat{y}^{\text{test}}$ given $x^{\text{test}}$, demonstration examples $\mathcal{D}_{\text{demo}}$, and auxiliary knowledge $\mathcal{A}$ derived from human trajectories. ARCTraj provides three structured components of $\mathcal{A}$, (1) selection biases, (2) color origins, and (3) shared intentions, that capture different stages of human reasoning and can be integrated into model learning.
• Figure 4: Distributions of human selection behavior in ARC tasks. Left: Selections are concentrated in compact shapes ranging from $1\times1$ to $3\times3$, with square- and bar-shaped regions dominating. Right: Most selections cover fewer than 20 pixels, reflecting a preference for local and perceptually salient regions.
• Figure 5: Uniqueness analysis of human reasoning trajectories. According to the left panel, most ARC tasks show low unique trajectory ratios, indicating that human solvers often converge on similar reasoning paths. The right panel shows a representative low-uniqueness task (https://o2arc.com/task/c0f76784; see Appendix \ref{['appendix:arc_task']}), where overlapping solution routes appear in the state-space graph.