Addressing and Visualizing Misalignments in Human Task-Solving Trajectories

TL;DR

This work addresses misalignments between user intentions and actions in ARC task trajectories by formalizing three misalignment types and introducing a unified framework with popular states, ideal actions, and ideal trajectories. It develops a Misalignment Detection Algorithm and an Intention Prediction Algorithm to align trajectories with inferred human intentions, then demonstrates that intention-aligned supervision improves sequential learning in AI models—achieving near 90% accuracy when combined with object-level features. The findings highlight that most trajectories are intention-aligned, but tool limitations and user unfamiliarity substantially contribute to inefficiencies, which can be mitigated by enhancing tool functionality and structured representations of intentions. Overall, the paper provides a systematic approach to trajectory-based learning that aligns AI reasoning with human problem-solving strategies, offering actionable insights for UI design, data annotation, and model training in trajectory-rich domains like ARC tasks.

Abstract

Understanding misalignments in human task-solving trajectories is crucial for enhancing AI models trained to closely mimic human reasoning. This study categorizes such misalignments into three types: (1) lack of functions to express intent, (2) inefficient action sequences, and (3) incorrect intentions that cannot solve the task. To address these issues, we first formalize and define these three misalignment types in a unified framework. We then propose a heuristic algorithm to detect misalignments in ARCTraj trajectories and analyze their impact hierarchically and quantitatively. We also present an intention estimation method based on our formalism that infers missing alignment between user actions and intentions. Through trajectory alignment, we experimentally demonstrate that AI models trained on human task-solving trajectories improve performance in mimicking human reasoning. Based on hierarchical analysis and experiments, we highlight the importance of trajectory-intention alignment and demonstrate the effectiveness of intention-aligned training.

Figures (12)

• Figure 1: An example of ARC task (Task 23b5c85d). The goal is to infer the rule from input-output pairs and generate the correct output grid for a new input. The correct answer is the $3 \times 3$ magenta-colored rectangle.
• Figure 2: A single state transition step in ARCLE lee2024arcle. An action transforms the current state $s_i$ into the next state $s_{i+1}$ through the transition function $f(s_i, a_i) = s_{i+1}$. In this example, the selected grids masked in white are shifted down by one row.
• Figure 3: Various trajectories for ARC Task 23b5c85d as shown in Fig. \ref{['fig:example_arc_task']}. (a) The ideal trajectory transitions directly between popular states with the shortest possible sequence of actions, perfectly representing user intentions. (b) Functional inadequacies influence the trajectory in tools, where the lack of a suitable action requires combining multiple actions, resulting in longer transitions between popular states. (c) A trajectory is caused by user unfamiliarity with tools, where redundant actions reflect inefficiencies despite a shorter ideal trajectory. (d) Trajectory reflects cognitive dissonance in users, where errors or misinterpretations prevent reaching the correct answer state, deviating from transitions between popular states.
• Figure 4: The state space graph of user trajectories from Fig. \ref{['fig:trajectory_types']}. The blue node represents the test problem state, the green node denotes the answer state, and the red nodes indicate states where incorrect answers were submitted. The thickness of nodes and edges reflects the frequency of occurrence for each respective state and action.
• Figure 5: A Venn diagram of misalignment types at the trajectory level, showing proportions of trajectories with different misalignment types and overlaps between them.